153 lines
5.8 KiB
Markdown
153 lines
5.8 KiB
Markdown
# Changelog
|
|
|
|
All notable changes to the Orderflow Backtest System are documented in this file.
|
|
|
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
|
|
## [Unreleased]
|
|
|
|
### Added
|
|
- Comprehensive documentation structure with module-specific guides
|
|
- Architecture Decision Records (ADRs) for major technical decisions
|
|
- CONTRIBUTING.md with development guidelines and standards
|
|
- Enhanced module documentation in `docs/modules/` directory
|
|
- Dependency documentation with security and performance considerations
|
|
|
|
### Changed
|
|
- Documentation structure reorganized to follow documentation standards
|
|
- Improved code documentation requirements with examples
|
|
- Enhanced testing guidelines with coverage requirements
|
|
|
|
## [2.0.0] - 2024-12-Present
|
|
|
|
### Added
|
|
- **Simplified Pipeline Architecture**: Streamlined SQLite → OHLC/Depth → JSON → Dash pipeline
|
|
- **JSON-based IPC**: Atomic file-based communication between processor and visualizer
|
|
- **Real-time Visualization**: Dash web application with 500ms polling updates
|
|
- **OHLC Aggregation**: Configurable time window aggregation with throttled updates
|
|
- **Orderbook Depth**: Real-time depth snapshots with top-N level management
|
|
- **OBI Metrics**: Order Book Imbalance calculation with candlestick visualization
|
|
- **Atomic JSON Operations**: Race-condition-free data exchange via temp files
|
|
- **CLI Orchestration**: Typer-based command interface with process management
|
|
- **Performance Optimizations**: Batch reading with optimized SQLite PRAGMA settings
|
|
|
|
### Changed
|
|
- **Architecture Simplification**: Removed complex repository/storage layers
|
|
- **Data Flow**: Direct streaming from database to visualization via JSON
|
|
- **Error Handling**: Graceful degradation with cached data fallbacks
|
|
- **Process Management**: Separate visualization process launched automatically
|
|
- **Memory Efficiency**: Bounded datasets prevent unlimited memory growth
|
|
|
|
### Technical Details
|
|
- **Database Access**: Read-only SQLite with immutable mode and mmap optimization
|
|
- **Batch Sizes**: BOOK_BATCH=2048, TRADE_BATCH=4096 for optimal performance
|
|
- **JSON Formats**: Standardized schemas for OHLC, depth, and metrics data
|
|
- **Chart Architecture**: Multi-subplot layout with shared time axis
|
|
- **IPC Files**: `ohlc_data.json`, `depth_data.json`, `metrics_data.json`
|
|
|
|
### Removed
|
|
- Complex metrics storage and repository patterns
|
|
- Strategy framework components
|
|
- In-memory snapshot retention
|
|
- Multi-database orchestration complexity
|
|
|
|
## [1.0.0] - Previous Version
|
|
|
|
### Features
|
|
- **Orderbook Reconstruction**: Build complete orderbooks from SQLite database files
|
|
- **Data Models**: Core structures for `OrderbookLevel`, `Trade`, `BookSnapshot`, `Book`
|
|
- **SQLite Repository**: Read-only data access for orderbook and trades data
|
|
- **Orderbook Parser**: Text parsing with price caching optimization
|
|
- **Storage Orchestration**: High-level facade for book building
|
|
- **Basic Visualization**: OHLC candlestick charts with Qt5Agg backend
|
|
- **Strategy Framework**: Basic strategy pattern with `DefaultStrategy`
|
|
- **CLI Interface**: Command-line application for date range processing
|
|
- **Test Suite**: Unit and integration tests
|
|
|
|
### Architecture
|
|
- **Repository Pattern**: Clean separation of data access logic
|
|
- **Dataclass Models**: Lightweight, type-safe data structures
|
|
- **Parser Optimization**: Price caching for performance
|
|
- **Modular Design**: Clear separation between components
|
|
|
|
---
|
|
|
|
## Migration Guide
|
|
|
|
### Upgrading from v1.0.0 to v2.0.0
|
|
|
|
#### Code Changes Required
|
|
|
|
1. **Strategy Constructor**
|
|
```python
|
|
# Before (v1.0.0)
|
|
strategy = DefaultStrategy("BTC-USDT", enable_visualization=True)
|
|
|
|
# After (v2.0.0)
|
|
strategy = DefaultStrategy("BTC-USDT")
|
|
visualizer = Visualizer(window_seconds=60, max_bars=500)
|
|
```
|
|
|
|
2. **Main Application Flow**
|
|
```python
|
|
# Before (v1.0.0)
|
|
strategy = DefaultStrategy(instrument, enable_visualization=True)
|
|
storage.build_booktick_from_db(db_path, db_date)
|
|
strategy.on_booktick(storage.book)
|
|
|
|
# After (v2.0.0)
|
|
strategy = DefaultStrategy(instrument)
|
|
visualizer = Visualizer(window_seconds=60, max_bars=500)
|
|
|
|
strategy.set_db_path(db_path)
|
|
visualizer.set_db_path(db_path)
|
|
storage.build_booktick_from_db(db_path, db_date)
|
|
strategy.on_booktick(storage.book)
|
|
visualizer.update_from_book(storage.book)
|
|
```
|
|
|
|
#### Database Migration
|
|
- **Automatic**: Metrics table created automatically on first run
|
|
- **No Data Loss**: Existing orderbook and trades data unchanged
|
|
- **Schema Addition**: New `metrics` table with indexes added to existing databases
|
|
|
|
#### Benefits of Upgrading
|
|
- **Memory Efficiency**: >70% reduction in memory usage
|
|
- **Performance**: Faster processing through persistent metrics storage
|
|
- **Enhanced Analysis**: Access to OBI and CVD financial indicators
|
|
- **Better Visualization**: Multi-chart display with synchronized time axis
|
|
- **Improved Architecture**: Cleaner separation of concerns
|
|
|
|
#### Testing Migration
|
|
```bash
|
|
# Verify upgrade compatibility
|
|
uv run pytest tests/test_main_integration.py -v
|
|
|
|
# Test new metrics functionality
|
|
uv run pytest tests/test_storage_metrics.py -v
|
|
|
|
# Validate visualization separation
|
|
uv run pytest tests/test_main_visualization.py -v
|
|
```
|
|
|
|
---
|
|
|
|
## Development Notes
|
|
|
|
### Performance Improvements
|
|
- **v2.0.0**: >70% memory reduction, batch processing, persistent storage
|
|
- **v1.0.0**: In-memory processing, real-time calculations
|
|
|
|
### Architecture Evolution
|
|
- **v2.0.0**: Streaming processing with metrics storage, separated visualization
|
|
- **v1.0.0**: Full snapshot retention, integrated visualization in strategies
|
|
|
|
### Testing Coverage
|
|
- **v2.0.0**: 27 tests across 6 files, integration and unit coverage
|
|
- **v1.0.0**: Basic unit tests for core components
|
|
|
|
---
|
|
|
|
*For detailed technical documentation, see [docs/](../docs/) directory.*
|