orderflow_backtest/docs/CHANGELOG.md

Changelog

All notable changes to the Orderflow Backtest System are documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Added

• Comprehensive documentation structure with module-specific guides
• Architecture Decision Records (ADRs) for major technical decisions
• CONTRIBUTING.md with development guidelines and standards
• Enhanced module documentation in docs/modules/ directory
• Dependency documentation with security and performance considerations

Changed

• Documentation structure reorganized to follow documentation standards
• Improved code documentation requirements with examples
• Enhanced testing guidelines with coverage requirements

[2.0.0] - 2024-12-Present

Added

• Simplified Pipeline Architecture: Streamlined SQLite → OHLC/Depth → JSON → Dash pipeline
• JSON-based IPC: Atomic file-based communication between processor and visualizer
• Real-time Visualization: Dash web application with 500ms polling updates
• OHLC Aggregation: Configurable time window aggregation with throttled updates
• Orderbook Depth: Real-time depth snapshots with top-N level management
• OBI Metrics: Order Book Imbalance calculation with candlestick visualization
• Atomic JSON Operations: Race-condition-free data exchange via temp files
• CLI Orchestration: Typer-based command interface with process management
• Performance Optimizations: Batch reading with optimized SQLite PRAGMA settings

Changed

• Architecture Simplification: Removed complex repository/storage layers
• Data Flow: Direct streaming from database to visualization via JSON
• Error Handling: Graceful degradation with cached data fallbacks
• Process Management: Separate visualization process launched automatically
• Memory Efficiency: Bounded datasets prevent unlimited memory growth

Technical Details

• Database Access: Read-only SQLite with immutable mode and mmap optimization
• Batch Sizes: BOOK_BATCH=2048, TRADE_BATCH=4096 for optimal performance
• JSON Formats: Standardized schemas for OHLC, depth, and metrics data
• Chart Architecture: Multi-subplot layout with shared time axis
• IPC Files: ohlc_data.json, depth_data.json, metrics_data.json

Removed

• Complex metrics storage and repository patterns
• Strategy framework components
• In-memory snapshot retention
• Multi-database orchestration complexity

[1.0.0] - Previous Version

Features

• Orderbook Reconstruction: Build complete orderbooks from SQLite database files
• Data Models: Core structures for OrderbookLevel, Trade, BookSnapshot, Book
• SQLite Repository: Read-only data access for orderbook and trades data
• Orderbook Parser: Text parsing with price caching optimization
• Storage Orchestration: High-level facade for book building
• Basic Visualization: OHLC candlestick charts with Qt5Agg backend
• Strategy Framework: Basic strategy pattern with DefaultStrategy
• CLI Interface: Command-line application for date range processing
• Test Suite: Unit and integration tests

Architecture

• Repository Pattern: Clean separation of data access logic
• Dataclass Models: Lightweight, type-safe data structures
• Parser Optimization: Price caching for performance
• Modular Design: Clear separation between components

---

Migration Guide

Upgrading from v1.0.0 to v2.0.0

Code Changes Required

1. Strategy Constructor

   # Before (v1.0.0)
   strategy = DefaultStrategy("BTC-USDT", enable_visualization=True)
   
   # After (v2.0.0)  
   strategy = DefaultStrategy("BTC-USDT")
   visualizer = Visualizer(window_seconds=60, max_bars=500)
   

2. Main Application Flow

   # Before (v1.0.0)
   strategy = DefaultStrategy(instrument, enable_visualization=True)
   storage.build_booktick_from_db(db_path, db_date)
   strategy.on_booktick(storage.book)
   
   # After (v2.0.0)
   strategy = DefaultStrategy(instrument)
   visualizer = Visualizer(window_seconds=60, max_bars=500)
   
   strategy.set_db_path(db_path)
   visualizer.set_db_path(db_path)
   storage.build_booktick_from_db(db_path, db_date)
   strategy.on_booktick(storage.book)
   visualizer.update_from_book(storage.book)
   

Database Migration

• Automatic: Metrics table created automatically on first run
• No Data Loss: Existing orderbook and trades data unchanged
• Schema Addition: New metrics table with indexes added to existing databases

Benefits of Upgrading

• Memory Efficiency: >70% reduction in memory usage
• Performance: Faster processing through persistent metrics storage
• Enhanced Analysis: Access to OBI and CVD financial indicators
• Better Visualization: Multi-chart display with synchronized time axis
• Improved Architecture: Cleaner separation of concerns

Testing Migration

# Verify upgrade compatibility
uv run pytest tests/test_main_integration.py -v

# Test new metrics functionality  
uv run pytest tests/test_storage_metrics.py -v

# Validate visualization separation
uv run pytest tests/test_main_visualization.py -v

---

Development Notes

Performance Improvements

• v2.0.0: >70% memory reduction, batch processing, persistent storage
• v1.0.0: In-memory processing, real-time calculations

Architecture Evolution

• v2.0.0: Streaming processing with metrics storage, separated visualization
• v1.0.0: Full snapshot retention, integrated visualization in strategies

Testing Coverage

• v2.0.0: 27 tests across 6 files, integration and unit coverage
• v1.0.0: Basic unit tests for core components

---

For detailed technical documentation, see docs/ directory.