orderflow_backtest/docs/CHANGELOG.md

Changelog

All notable changes to the Orderflow Backtest System are documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[2.0.0] - 2024-Current

Added

• OBI Metrics Calculation: Order Book Imbalance calculation with formula (Vb - Va) / (Vb + Va)
• CVD Metrics Calculation: Cumulative Volume Delta with incremental calculation and reset functionality
• Persistent Metrics Storage: SQLite-based storage for calculated metrics to avoid recalculation
• Memory Optimization: >70% reduction in peak memory usage through streaming processing
• Enhanced Visualization: Multi-subplot charts with OHLC, Volume, OBI, and CVD displays
• MetricCalculator Class: Static methods for financial metrics computation
• Batch Processing: High-performance batch inserts (1000 records per operation)
• Time-Range Queries: Efficient metrics retrieval for specified time periods
• Strategy Enhancement: Metrics analysis capabilities in DefaultStrategy
• Comprehensive Testing: 27 tests across 6 test files with full integration coverage

Changed

• Storage Architecture: Modified Storage.build_booktick_from_db() to integrate metrics calculation
• Visualization Separation: Moved visualization from strategy to main application for better separation of concerns
• Strategy Interface: Simplified DefaultStrategy constructor (removed enable_visualization parameter)
• Main Application Flow: Enhanced orchestration with per-database visualization updates
• Database Schema: Auto-creation of metrics table with proper indexes and foreign key constraints
• Memory Management: Stream processing instead of keeping full snapshot history

Improved

• Performance: Batch database operations and optimized SQLite PRAGMAs
• Scalability: Support for months to years of high-frequency trading data
• Code Quality: All functions <50 lines, all files <250 lines
• Documentation: Comprehensive module and API documentation
• Error Handling: Graceful degradation and comprehensive logging
• Type Safety: Full type annotations throughout codebase

Technical Details

• New Tables: metrics table with indexes on timestamp and snapshot_id
• New Models: Metric dataclass for calculated values
• Processing Pipeline: Snapshot → Calculate → Store → Discard workflow
• Query Interface: Time-range based metrics retrieval
• Visualization Layout: 4-subplot layout with shared time axis

[1.0.0] - Previous Version

Features

• Orderbook Reconstruction: Build complete orderbooks from SQLite database files
• Data Models: Core structures for OrderbookLevel, Trade, BookSnapshot, Book
• SQLite Repository: Read-only data access for orderbook and trades data
• Orderbook Parser: Text parsing with price caching optimization
• Storage Orchestration: High-level facade for book building
• Basic Visualization: OHLC candlestick charts with Qt5Agg backend
• Strategy Framework: Basic strategy pattern with DefaultStrategy
• CLI Interface: Command-line application for date range processing
• Test Suite: Unit and integration tests

Architecture

• Repository Pattern: Clean separation of data access logic
• Dataclass Models: Lightweight, type-safe data structures
• Parser Optimization: Price caching for performance
• Modular Design: Clear separation between components

---

Migration Guide

Upgrading from v1.0.0 to v2.0.0

Code Changes Required

1. Strategy Constructor

   # Before (v1.0.0)
   strategy = DefaultStrategy("BTC-USDT", enable_visualization=True)
   
   # After (v2.0.0)  
   strategy = DefaultStrategy("BTC-USDT")
   visualizer = Visualizer(window_seconds=60, max_bars=500)
   

2. Main Application Flow

   # Before (v1.0.0)
   strategy = DefaultStrategy(instrument, enable_visualization=True)
   storage.build_booktick_from_db(db_path, db_date)
   strategy.on_booktick(storage.book)
   
   # After (v2.0.0)
   strategy = DefaultStrategy(instrument)
   visualizer = Visualizer(window_seconds=60, max_bars=500)
   
   strategy.set_db_path(db_path)
   visualizer.set_db_path(db_path)
   storage.build_booktick_from_db(db_path, db_date)
   strategy.on_booktick(storage.book)
   visualizer.update_from_book(storage.book)
   

Database Migration

• Automatic: Metrics table created automatically on first run
• No Data Loss: Existing orderbook and trades data unchanged
• Schema Addition: New metrics table with indexes added to existing databases

Benefits of Upgrading

• Memory Efficiency: >70% reduction in memory usage
• Performance: Faster processing through persistent metrics storage
• Enhanced Analysis: Access to OBI and CVD financial indicators
• Better Visualization: Multi-chart display with synchronized time axis
• Improved Architecture: Cleaner separation of concerns

Testing Migration

# Verify upgrade compatibility
uv run pytest tests/test_main_integration.py -v

# Test new metrics functionality  
uv run pytest tests/test_storage_metrics.py -v

# Validate visualization separation
uv run pytest tests/test_main_visualization.py -v

---

Development Notes

Performance Improvements

• v2.0.0: >70% memory reduction, batch processing, persistent storage
• v1.0.0: In-memory processing, real-time calculations

Architecture Evolution

• v2.0.0: Streaming processing with metrics storage, separated visualization
• v1.0.0: Full snapshot retention, integrated visualization in strategies

Testing Coverage

• v2.0.0: 27 tests across 6 files, integration and unit coverage
• v1.0.0: Basic unit tests for core components

---

For detailed technical documentation, see docs/ directory.