Simon/orderflow_backtest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

WIP UI rework with qt6

Browse Source
This commit is contained in:
Simon Moisy
2025-09-10 15:39:16 +08:00
parent 36385af6f3
commit ebf232317c
63 changed files with 4005 additions and 5221 deletions
Download Patch File Download Diff File Expand all files Collapse all files

76
tasks/prd-cumulative-volume-delta.md Normal file
View File

@@ -0,0 +1,76 @@
## Cumulative Volume Delta (CVD) – Product Requirements Document
### 1) Introduction / Overview
- Compute and visualize Cumulative Volume Delta (CVD) from trade data processed by `OHLCProcessor.process_trades`, aligned to the existing OHLC bar cadence.
- CVD is defined as the cumulative sum of volume delta, where volume delta = buy_volume - sell_volume per trade.
- Trade classification: `side == "buy"` → positive volume delta, `side == "sell"` → negative volume delta.
- Persist CVD time series as scalar values per window to `metrics_data.json` and render a CVD line chart beneath the current OBI subplot in the Dash UI.
### 2) Goals
- Compute volume delta from individual trades using the `side` field in the Trade dataclass.
- Accumulate CVD across all processed trades (no session resets initially).
- Aggregate CVD into window-aligned scalar values per `window_seconds`.
- Extend `metrics_data.json` schema to include CVD values alongside existing OBI data.
- Add a CVD line chart subplot beneath OBI in the main chart, sharing the time axis.
- Throttle intra-window upserts of CVD values using the same approach/frequency as current OHLC throttling; always write on window close.
### 3) User Stories
- As a researcher, I want CVD computed from actual trade data so I can assess buying/selling pressure over time.
- As an analyst, I want CVD stored per time window so I can correlate it with price movements and OBI patterns.
- As a developer, I want cumulative CVD values so I can analyze long-term directional bias in volume flow.
### 4) Functional Requirements
1. Inputs and Definitions
- Compute volume delta on every trade in `OHLCProcessor.process_trades`:
- If `trade.side == "buy"` → `volume_delta = +trade.size`
- If `trade.side == "sell"` → `volume_delta = -trade.size`
- If `trade.side` is neither "buy" nor "sell" → `volume_delta = 0` (log warning)
- Accumulate into running CVD: `self.cvd_cumulative += volume_delta`
2. Windowing & Aggregation
- Use the same `window_seconds` boundary as OHLC bars; window anchor is derived from the trade timestamp.
- Store CVD value at window boundaries (end-of-window CVD snapshot).
- On window rollover, capture the current `self.cvd_cumulative` value for that window.
3. Persistence
- Extend `metrics_data.json` schema from `[timestamp, obi_open, obi_high, obi_low, obi_close]` to `[timestamp, obi_open, obi_high, obi_low, obi_close, cvd_value]`.
- Update `viz_io.py` functions to handle the new 6-element schema.
- Keep only the last 1000 rows.
- Upsert intra-window CVD values periodically (throttled, matching OHLC's approach) and always write on window close.
4. Visualization
- Read extended `metrics_data.json` in the Dash app with the same tolerant JSON reading/caching approach.
- Extend the main figure to a fourth row for CVD line chart beneath OBI, sharing the x-axis.
- Style CVD as a line chart with appropriate color (distinct from OHLC/Volume/OBI) and add a zero baseline.
5. Performance & Correctness
- CVD compute happens on every trade; I/O is throttled to maintain UI responsiveness.
- Use existing logging and error handling patterns; must not crash if metrics JSON is temporarily unreadable.
- Handle backward compatibility: if existing `metrics_data.json` has 5-element rows, treat missing CVD as 0.
6. Testing
- Unit tests for volume delta calculation with "buy", "sell", and invalid side values.
- Unit tests for CVD accumulation across multiple trades and window boundaries.
- Integration test: fixture trades produce correct CVD progression in `metrics_data.json`.
### 5) Non-Goals
- No CVD reset functionality (will be implemented later).
- No additional derived CVD metrics (e.g., CVD rate of change, normalized CVD).
- No database persistence for CVD; JSON IPC only.
- No strategy/signal changes based on CVD.
### 6) Design Considerations
- Implement CVD calculation in `OHLCProcessor.process_trades` alongside existing OHLC aggregation.
- Extend `viz_io.py` metrics functions to support 6-element schema while maintaining backward compatibility.
- Add CVD state tracking: `self.cvd_cumulative`, `self.cvd_window_value` per window.
- Follow the same throttling pattern as OBI metrics for consistency.
### 7) Technical Considerations
- Add CVD computation in the trade processing loop within `OHLCProcessor.process_trades`.
- Extend `upsert_metric_bar` and `add_metric_bar` functions to accept optional `cvd_value` parameter.
- Handle schema migration gracefully: read existing 5-element rows, append 0.0 for missing CVD.
- Use the same window alignment as trades (based on trade timestamp, not orderbook timestamp).
### 8) Success Metrics
- `metrics_data.json` present with valid 6-element rows during processing.
- CVD subplot updates smoothly and aligns with OHLC window timestamps.
- CVD increases during buy-heavy periods, decreases during sell-heavy periods.
- No noticeable performance regression in trade processing or UI responsiveness.
### 9) Open Questions
- None; CVD computation approach confirmed using trade.side field. Schema extension approach confirmed for metrics_data.json.

208
tasks/prd-interactive-visualizer.md
View File

@@ -1,208 +0,0 @@
# PRD: Interactive Visualizer with Plotly + Dash
## Introduction/Overview
The current orderflow backtest system uses a static matplotlib-based visualizer that displays OHLC candlesticks, volume bars, Order Book Imbalance (OBI), and Cumulative Volume Delta (CVD) charts. This PRD outlines the development of a new interactive visualization system using Plotly + Dash that will provide real-time interactivity, detailed data inspection, and enhanced user experience for cryptocurrency trading analysis.
The goal is to replace the static visualization with a professional, web-based interactive dashboard that allows traders to explore orderbook metrics with precision and flexibility.
## Goals
1. **Replace Static Visualization**: Create a new `InteractiveVisualizer` class using Plotly + Dash
2. **Enable Cross-Chart Interactivity**: Implement synchronized zooming, panning, and time range selection across all charts
3. **Provide Precision Navigation**: Add crosshair cursor with vertical line indicator across all charts
4. **Display Contextual Information**: Show detailed metrics in a side panel when hovering over data points
5. **Support Multiple Time Granularities**: Allow users to adjust time resolution dynamically
6. **Maintain Performance**: Handle large datasets (months of data) with smooth interactions
7. **Preserve Integration**: Seamlessly integrate with existing metrics storage and data processing pipeline
## User Stories
### Primary Use Cases
- **US-1**: As a trader, I want to zoom into specific time periods across all charts simultaneously so that I can analyze market behavior during critical moments
- **US-2**: As a trader, I want to see a vertical crosshair line that spans all charts so that I can precisely align data points across OHLC, volume, OBI, and CVD metrics
- **US-3**: As a trader, I want to hover over any data point and see detailed information in a side panel so that I can inspect exact values without cluttering the charts
- **US-4**: As a trader, I want to pan through historical data smoothly so that I can explore different time periods efficiently
- **US-5**: As a trader, I want to reset CVD calculations from a selected point in time so that I can analyze cumulative volume delta from specific market events
### Secondary Use Cases
- **US-6**: As a trader, I want to adjust time granularity (1min, 5min, 1hour) so that I can view data at different resolutions
- **US-7**: As a trader, I want navigation controls (reset zoom, home button) so that I can quickly return to full data view
- **US-8**: As a trader, I want to select custom time ranges so that I can focus analysis on specific market sessions
## Functional Requirements
### Core Interactive Features
1. **F1**: The system must provide synchronized zooming across all four charts (OHLC, Volume, OBI, CVD)
2. **F2**: The system must provide synchronized panning across all four charts with shared X-axis
3. **F3**: The system must display a vertical crosshair line that spans all charts and follows mouse cursor
4. **F4**: The system must show detailed hover information for each chart type:
- OHLC: timestamp, open, high, low, close, spread
- Volume: timestamp, total volume, buy/sell breakdown if available
- OBI: timestamp, OBI value, bid volume, ask volume, imbalance percentage
- CVD: timestamp, CVD value, volume delta, cumulative change
### User Interface Requirements
5. **F5**: The system must display charts in a 4-row layout with shared X-axis (OHLC on top, Volume, OBI, CVD at bottom)
6. **F6**: The system must provide a side panel on the right displaying detailed information for the current cursor position
7. **F7**: The system must include navigation controls:
- Zoom in/out buttons
- Reset zoom button
- Home view button
- Time range selector
8. **F8**: The system must provide time granularity controls (1min, 5min, 15min, 1hour, 6hour)
### Data Integration Requirements
9. **F9**: The system must integrate with existing `SQLiteOrderflowRepository` for metrics data loading
10. **F10**: The system must support loading data from multiple database files seamlessly
11. **F11**: The system must maintain the existing `set_db_path()` and `update_from_book()` interface for compatibility
12. **F12**: The system must calculate OHLC bars from snapshots with configurable time windows
### Performance Requirements
13. **F13**: The system must render charts with <2 second initial load time for datasets up to 1 million data points
14. **F14**: The system must provide smooth zooming and panning interactions with <100ms response time
15. **F15**: The system must efficiently update hover information with <50ms latency
### CVD Reset Functionality
16. **F16**: The system must allow users to click on any point in the CVD chart to reset cumulative calculation from that timestamp
17. **F17**: The system must visually indicate CVD reset points with markers or annotations
18. **F18**: The system must recalculate and redraw CVD values from the reset point forward
## Non-Goals (Out of Scope)
1. **Advanced Drawing Tools**: Trend lines, Fibonacci retracements, or annotation tools
2. **Multiple Instrument Support**: Multi-symbol comparison or overlay charts
3. **Real-time Streaming**: Live data updates or WebSocket integration
4. **Export Functionality**: Chart export to PNG/PDF or data export to CSV
5. **User Authentication**: User accounts, saved layouts, or personalization
6. **Mobile Optimization**: Touch interfaces or responsive mobile design
7. **Advanced Indicators**: Technical analysis indicators beyond OBI/CVD
8. **Alert System**: Price alerts, threshold notifications, or automated signals
## Design Considerations
### Chart Layout
- **Layout**: 4-row subplot layout with 80% chart area, 20% side panel
- **Color Scheme**: Professional dark theme with customizable colors
- **Typography**: Clear, readable fonts optimized for financial data
- **Responsive Design**: Adaptable to different screen sizes (desktop focus)
### Side Panel Design
```
┌─────────────────┐
│ Current Data │
├─────────────────┤
│ Time: 16:30:45 │
│ Price: $50,123 │
│ Volume: 1,234 │
│ OBI: 0.234 │
│ CVD: -123.45 │
├─────────────────┤
│ Controls │
│ [Reset CVD] │
│ [Zoom Reset] │
│ [Time Range ▼] │
│ [Granularity ▼] │
└─────────────────┘
```
### Navigation Controls
- **Zoom**: Mouse wheel, zoom box selection, zoom buttons
- **Pan**: Click and drag, arrow keys, scroll bars
- **Reset**: Double-click to auto-scale, reset button to full view
- **Selection**: Click and drag for time range selection
## Technical Considerations
### Architecture Changes
- **New Class**: `InteractiveVisualizer` class separate from existing `Visualizer`
- **Dependencies**: Add `dash`, `plotly`, `dash-bootstrap-components` to requirements
- **Web Server**: Dash development server for local deployment
- **Data Flow**: Maintain existing metrics loading pipeline, adapt to Plotly data structures
### Integration Points
```python
# Maintain existing interface for compatibility
class InteractiveVisualizer:
def set_db_path(self, db_path: Path) -> None
def update_from_book(self, book: Book) -> None
def show(self) -> None # Launch Dash server instead of plt.show()
```
### Data Structure Adaptation
- **OHLC Data**: Convert bars to Plotly candlestick format
- **Metrics Data**: Transform to Plotly time series format
- **Memory Management**: Implement data decimation for large datasets
- **Caching**: Cache processed data to improve interaction performance
### Technology Stack
- **Frontend**: Dash + Plotly.js for charts
- **Backend**: Python Dash server with existing data pipeline
- **Styling**: Dash Bootstrap Components for professional UI
- **Data Processing**: Pandas for efficient data manipulation
## Success Metrics
### User Experience Metrics
1. **Interaction Responsiveness**: 95% of zoom/pan operations complete within 100ms
2. **Data Precision**: 100% accuracy in crosshair positioning and hover data display
3. **Navigation Efficiency**: Users can navigate to specific time periods 3x faster than static charts
### Technical Performance Metrics
4. **Load Time**: Initial chart rendering completes within 2 seconds for 500k data points
5. **Memory Usage**: Interactive visualizer uses <150% memory compared to static version
6. **Error Rate**: <1% interaction failures or display errors during normal usage
### Feature Adoption Metrics
7. **Feature Usage**: CVD reset functionality used in >30% of analysis sessions
8. **Time Range Analysis**: Custom time range selection used in >50% of sessions
9. **Granularity Changes**: Time resolution adjustment used in >40% of sessions
## Implementation Priority
### Phase 1: Core Interactive Charts (High Priority)
- Basic Plotly + Dash setup
- 4-chart layout with synchronized axes
- Basic zoom, pan, and crosshair functionality
- Integration with existing data pipeline
### Phase 2: Enhanced Interactivity (High Priority)
- Side panel with hover information
- Navigation controls and buttons
- Time granularity selection
- CVD reset functionality
### Phase 3: Performance Optimization (Medium Priority)
- Large dataset handling
- Interaction performance tuning
- Memory usage optimization
- Error handling and edge cases
### Phase 4: Polish and UX (Medium Priority)
- Professional styling and themes
- Enhanced navigation controls
- Time range selection tools
- User experience refinements
## Open Questions
1. **Deployment Method**: Should the interactive visualizer run as a local Dash server or be deployable as a standalone web application?
2. **Data Decimation Strategy**: How should the system handle datasets with millions of points while maintaining interactivity? Should it implement automatic decimation based on zoom level?
3. **CVD Reset Persistence**: Should CVD reset points be saved to the database or only exist in the current session?
4. **Multiple Database Sessions**: How should the interactive visualizer handle switching between different database files during the same session?
5. **Backward Compatibility**: Should the system maintain both static and interactive visualizers, or completely replace the matplotlib implementation?
6. **Configuration Management**: How should users configure default time granularities, color schemes, and layout preferences?
7. **Performance Baselines**: What are the acceptable performance thresholds for different dataset sizes and interaction types?
---
**Document Version**: 1.0
**Created**: Current Date
**Target Audience**: Junior Developer
**Estimated Implementation**: 3-4 weeks for complete feature set

71
tasks/prd-order-book-imbalance.md Normal file
View File

@@ -0,0 +1,71 @@
## Order Book Imbalance (OBI) – Product Requirements Document
### 1) Introduction / Overview
- Compute and visualize Order Book Imbalance (OBI) from the in-memory order book maintained by `OHLCProcessor`, aligned to the existing OHLC bar cadence.
- OBI is defined as raw `B - A`, where `B` is total bid size and `A` is total ask size.
- Persist an OBI time series as OHLC-style bars to `metrics_data.json` and render an OBI candlestick chart beneath the current Volume subplot in the Dash UI.
### 2) Goals
- Compute OBI from the full in-memory aggregated book (all bid/ask levels) on every order book update.
- Aggregate OBI into OHLC-style bars per `window_seconds`.
- Persist OBI bars to `metrics_data.json` with atomic writes and a rolling retention of 1000 rows.
- Add an OBI candlestick subplot (blue-toned) beneath Volume in the main chart, sharing the time axis.
- Throttle intra-window upserts of OBI bars using the same approach/frequency as current OHLC throttling; always write on window close.
### 3) User Stories
- As a researcher, I want OBI computed from the entire book so I can assess true depth imbalance.
- As an analyst, I want OBI stored per time window as candlesticks so I can compare it with price/volume behavior.
- As a developer, I want raw OBI values so I can analyze absolute imbalance patterns.
### 4) Functional Requirements
1. Inputs and Definitions
- Compute on every order book update using the complete in-memory book:
- `B = sum(self._book_bids.values())`
- `A = sum(self._book_asks.values())`
- `OBI = B - A`
- Edge case: if both sides are empty → `OBI = 0`.
2. Windowing & Aggregation
- Use the same `window_seconds` boundary as OHLC bars; window anchor is derived from the order book update timestamp.
- Maintain OBI OHLC per window: `obi_open`, `obi_high`, `obi_low`, `obi_close`.
- On window rollover, finalize and persist the bar.
3. Persistence
- Introduce `metrics_data.json` (co-located with other IPC files) with atomic writes.
- Schema: list of fixed-length rows
- `[timestamp_ms, obi_open, obi_high, obi_low, obi_close]`
- Keep only the last 1000 rows.
- Upsert intra-window bars periodically (throttled, matching OHLC’s approach) and always write on window close.
4. Visualization
- Read `metrics_data.json` in the Dash app with the same tolerant JSON reading/caching approach as other IPC files.
- Extend the main figure to a third row for OBI candlesticks beneath Volume, sharing the x-axis.
- Style OBI candlesticks in blue tones (distinct increasing/decreasing shades) and add a zero baseline.
5. Performance & Correctness
- OBI compute happens on every order book update; I/O is throttled to maintain UI responsiveness.
- Use existing logging and error handling patterns; must not crash if metrics JSON is temporarily unreadable.
6. Testing
- Unit tests for OBI on symmetric, empty, and imbalanced books; intra-window aggregation; window rollover.
- Integration test: fixture DB produces `metrics_data.json` aligned with OHLC bars, valid schema/lengths.
### 5) Non-Goals
- No additional derived metrics; keep only raw OBI values for maximum flexibility.
- No database persistence for metrics; JSON IPC only.
- No strategy/signal changes.
### 6) Design Considerations
- Reuse `OHLCProcessor` in-memory book (`_book_bids`, `_book_asks`).
- Introduce new metrics IO helpers in `viz_io.py` mirroring existing OHLC IO (atomic write, rolling trim, upsert).
- Keep `metrics_data.json` separate from `ohlc_data.json` to avoid schema churn.
### 7) Technical Considerations
- Implement OBI compute and aggregation inside `OHLCProcessor.update_orderbook` after applying partial updates.
- Throttle intra-window upserts with the same cadence concept as OHLC; on window close always persist.
- Add a finalize path to persist the last OBI bar.
### 8) Success Metrics
- `metrics_data.json` present with valid rows during processing.
- OBI subplot updates smoothly and aligns with OHLC window timestamps.
- OBI ≈ 0 for symmetric books; correct sign for imbalanced cases; no noticeable performance regression.
### 9) Open Questions
- None; cadence confirmed to match OHLC throttling. Styling: blue tones for OBI candlesticks.

190
tasks/prd-pyside6-pyqtgraph-migration.md Normal file
View File

@@ -0,0 +1,190 @@
# Product Requirements Document: Migration from Dash/Plotly to PySide6/PyQtGraph
## Introduction/Overview
This PRD outlines the complete migration of the orderflow backtest visualization system from the current Dash/Plotly web-based implementation to a native desktop application using PySide6 and PyQtGraph. The migration addresses critical issues with the current implementation including async problems, debugging difficulties, performance bottlenecks, and data handling inefficiencies.
The goal is to create a robust, high-performance desktop application that provides better control over the codebase, eliminates current visualization bugs (particularly the CVD graph display issue), and enables future real-time trading strategy monitoring capabilities.
## Goals
1. **Eliminate Current Technical Issues**
- Resolve async-related problems causing visualization failures
- Fix CVD graph display issues that persist despite correct-looking code
- Enable proper debugging capabilities with breakpoint support
- Improve overall application performance and responsiveness
2. **Improve Development Experience**
- Gain better control over the codebase through native Python implementation
- Reduce dependency on intermediate file-based data exchange
- Simplify the development and debugging workflow
- Establish a foundation for future real-time capabilities
3. **Maintain and Enhance Visualization Capabilities**
- Preserve all existing chart types and interactions
- Improve performance for granular dataset handling
- Prepare infrastructure for real-time data streaming
- Enhance user experience through native desktop interface
## User Stories
1. **As a trading strategy developer**, I want to visualize OHLC data with volume, OBI, and CVD indicators in a single, synchronized view so that I can analyze market behavior patterns effectively.
2. **As a data analyst**, I want to zoom, pan, and select specific time ranges on charts so that I can focus on relevant market periods for detailed analysis.
3. **As a system developer**, I want to debug visualization issues with breakpoints and proper debugging tools so that I can identify and fix problems efficiently.
4. **As a performance-conscious user**, I want smooth chart rendering and interactions even with large, granular datasets so that my analysis workflow is not interrupted by lag or freezing.
5. **As a future trading system operator**, I want a foundation that can handle real-time data updates so that I can monitor live trading strategies effectively.
## Functional Requirements
### Core Visualization Components
1. **Main Chart Window**
- The system must display OHLC candlestick charts in a primary plot area
- The system must allow customizable time window selection for OHLC display
- The system must synchronize all chart components to the same time axis
2. **Integrated Indicator Charts**
- The system must display Volume bars below the OHLC chart
- The system must display Order Book Imbalance (OBI) indicator
- The system must display Cumulative Volume Delta (CVD) indicator
- All indicators must share the same X-axis as the OHLC chart
3. **Depth Chart Visualization**
- The system must display order book depth at selected time snapshots
- The system must update depth visualization based on time selection
- The system must provide clear bid/ask visualization
### User Interaction Features
4. **Chart Navigation**
- The system must support zoom in/out functionality across all charts
- The system must allow panning across time ranges
- The system must provide time range selection capabilities
- The system must support rectangle selection for detailed analysis
5. **Data Inspection**
- The system must display mouseover information for all chart elements
- The system must show precise values for OHLC, volume, OBI, and CVD data points
- The system must provide crosshair functionality for precise data reading
### Technical Architecture
6. **Application Framework**
- The system must be built using PySide6 for the GUI framework
- The system must use PyQtGraph for all chart rendering and interactions
- The system must implement a native desktop application architecture
7. **Data Integration**
- The system must integrate with existing data processing modules (metrics_calculator, ohlc_processor, orderbook_manager)
- The system must eliminate dependency on intermediate JSON files for data display
- The system must support direct in-memory data transfer between processing and visualization
8. **Performance Requirements**
- The system must handle granular datasets efficiently without UI blocking
- The system must provide smooth chart interactions (zoom, pan, selection)
- The system must render updates in less than 100ms for typical dataset sizes
### Development and Debugging
9. **Code Quality**
- The system must be fully debuggable with standard Python debugging tools
- The system must follow the existing project architecture patterns
- The system must maintain clean separation between data processing and visualization
## Non-Goals (Out of Scope)
1. **Web Interface Maintenance** - The existing Dash/Plotly implementation will be completely replaced, not maintained in parallel
2. **Backward Compatibility** - No requirement to maintain compatibility with existing Dash/Plotly components or web-based deployment
3. **Multi-Platform Distribution** - Initial focus on development environment only, not packaging for distribution
4. **Real-Time Implementation** - While the architecture should support future real-time capabilities, the initial migration will focus on historical data visualization
5. **Advanced Chart Types** - Only migrate existing chart types; new visualization features are out of scope for this migration
## Design Considerations
### User Interface Layout
- **Main Window Structure**: Primary chart area with integrated indicators below
- **Control Panel**: Side panel or toolbar for time range selection and chart configuration
- **Status Bar**: Display current data range, loading status, and performance metrics
- **Menu System**: File operations, view options, and application settings
### PyQtGraph Integration
- **Plot Organization**: Use PyQtGraph's PlotWidget for main charts with linked axes
- **Custom Plot Items**: Implement custom plot items for OHLC candlesticks and depth visualization
- **Performance Optimization**: Utilize PyQtGraph's fast plotting capabilities for large datasets
### Data Flow Architecture
- **Direct Memory Access**: Replace JSON file intermediates with direct Python object passing
- **Lazy Loading**: Implement efficient data loading strategies for large time ranges
- **Caching Strategy**: Cache processed data to improve navigation performance
## Technical Considerations
### Dependencies and Integration
- **PySide6**: Main GUI framework, provides native desktop capabilities
- **PyQtGraph**: High-performance plotting library, optimized for real-time data
- **Existing Modules**: Maintain integration with metrics_calculator.py, ohlc_processor.py, orderbook_manager.py
- **Database Integration**: Continue using existing SQLite database through db_interpreter.py
### Migration Strategy (Iterative Implementation)
- **Phase 1**: Basic PySide6 window with single PyQtGraph plot
- **Phase 2**: OHLC candlestick chart implementation
- **Phase 3**: Volume, OBI, and CVD indicator integration
- **Phase 4**: Depth chart implementation
- **Phase 5**: User interaction features (zoom, pan, selection)
- **Phase 6**: Data integration and performance optimization
### Performance Considerations
- **Memory Management**: Efficient data structure handling for large datasets
- **Rendering Optimization**: Use PyQtGraph's ViewBox and plotting optimizations
- **Thread Safety**: Proper handling of data processing in background threads
- **Resource Cleanup**: Proper cleanup of chart objects and data structures
## Success Metrics
### Technical Success Criteria
1. **Bug Resolution**: CVD graph displays correctly and all existing visualization bugs are resolved
2. **Performance Improvement**: Chart interactions respond within 100ms for typical datasets
3. **Debugging Capability**: Developers can set breakpoints and debug visualization code effectively
4. **Data Handling**: Elimination of intermediate JSON files reduces data transfer overhead by 50%
### User Experience Success Criteria
1. **Feature Parity**: All existing chart types and interactions are preserved and functional
2. **Responsiveness**: Application feels more responsive than the current Dash implementation
3. **Stability**: No crashes or freezing during normal chart operations
4. **Visual Quality**: Charts render clearly with proper scaling and anti-aliasing
### Development Success Criteria
1. **Code Maintainability**: New codebase follows established project patterns and is easier to maintain
2. **Development Velocity**: Future visualization features can be implemented more quickly
3. **Testing Capability**: Comprehensive testing can be performed with proper debugging tools
4. **Architecture Foundation**: System is ready for future real-time data integration
## Open Questions
1. **Data Loading Strategy**: Should we implement progressive loading for very large datasets, or rely on existing data chunking mechanisms?
2. **Configuration Management**: How should chart configuration and user preferences be stored and managed in the desktop application?
3. **Error Handling**: What specific error handling and user feedback mechanisms should be implemented for data loading and processing failures?
4. **Performance Monitoring**: Should we include built-in performance monitoring and profiling tools in the application?
5. **Future Real-Time Integration**: What specific interface patterns should be established now to facilitate future real-time data streaming integration?
## Implementation Approach
This migration will follow the iterative development workflow with explicit approval checkpoints between each phase. Each implementation phase will be:
- Limited to manageable scope (≤250 lines per module)
- Tested immediately after implementation
- Integrated with existing data processing modules
- Validated for performance and functionality before proceeding to the next phase
The implementation will begin with basic PySide6 application structure and progressively add PyQtGraph visualization capabilities while maintaining integration with the existing data processing pipeline.

74
tasks/tasks-prd-interactive-visualizer.md
View File

@@ -1,74 +0,0 @@
# Tasks: Interactive Visualizer with Plotly + Dash
## Relevant Files
- `interactive_visualizer.py` - Main InteractiveVisualizer class implementing Plotly + Dash interface
- `tests/test_interactive_visualizer.py` - Unit tests for InteractiveVisualizer class
- `dash_app.py` - Dash application setup and layout configuration
- `tests/test_dash_app.py` - Unit tests for Dash application components
- `dash_callbacks.py` - Dash callback functions for interactivity and data updates
- `tests/test_dash_callbacks.py` - Unit tests for callback functions
- `dash_components.py` - Custom Dash components for side panel and controls
- `tests/test_dash_components.py` - Unit tests for custom components
- `data_adapters.py` - Data transformation utilities for Plotly format conversion
- `tests/test_data_adapters.py` - Unit tests for data adapter functions
- `pyproject.toml` - Updated dependencies including dash, plotly, dash-bootstrap-components
- `main.py` - Updated to support both static and interactive visualizer options
### Notes
- Unit tests should be placed in the `tests/` directory following existing project structure
- Use `uv run pytest [optional/path/to/test/file]` to run tests following project conventions
- Dash server will run locally for development, accessible via browser at http://127.0.0.1:8050
- Maintain backward compatibility with existing matplotlib visualizer
## Tasks
- [ ] 1.0 Setup Plotly + Dash Infrastructure and Dependencies
- [ ] 1.1 Add dash, plotly, and dash-bootstrap-components to pyproject.toml dependencies
- [ ] 1.2 Install and verify new dependencies with uv sync
- [ ] 1.3 Create basic dash_app.py with minimal Dash application setup
- [ ] 1.4 Verify Dash server can start and serve a basic "Hello World" page
- [ ] 1.5 Create project structure for interactive visualizer modules
- [ ] 2.0 Create Core Interactive Chart Layout with Synchronized Axes
- [ ] 2.1 Design 4-subplot layout using plotly.subplots.make_subplots with shared X-axis
- [ ] 2.2 Implement OHLC candlestick chart using plotly.graph_objects.Candlestick
- [ ] 2.3 Implement Volume bar chart using plotly.graph_objects.Bar
- [ ] 2.4 Implement OBI line chart using plotly.graph_objects.Scatter
- [ ] 2.5 Implement CVD line chart using plotly.graph_objects.Scatter
- [ ] 2.6 Configure synchronized zooming and panning across all subplots
- [ ] 2.7 Add vertical crosshair functionality spanning all charts
- [ ] 2.8 Apply professional dark theme and styling to charts
- [ ] 3.0 Implement Data Integration and Processing Pipeline
- [ ] 3.1 Create InteractiveVisualizer class maintaining set_db_path() and update_from_book() interface
- [ ] 3.2 Implement data_adapters.py for converting Book/Metric data to Plotly format
- [ ] 3.3 Create OHLC data transformation from existing bar calculation logic
- [ ] 3.4 Create metrics data transformation for OBI and CVD time series
- [ ] 3.5 Implement volume data aggregation and formatting
- [ ] 3.6 Add data caching mechanism for improved performance
- [ ] 3.7 Integrate with existing SQLiteOrderflowRepository for metrics loading
- [ ] 3.8 Handle multiple database file support seamlessly
- [ ] 4.0 Build Interactive Features and Navigation Controls
- [ ] 4.1 Implement zoom in/out functionality with mouse wheel and buttons
- [ ] 4.2 Implement pan functionality with click and drag
- [ ] 4.3 Add reset zoom and home view buttons
- [ ] 4.4 Create time range selector component for custom period selection
- [ ] 4.5 Implement time granularity controls (1min, 5min, 15min, 1hour, 6hour)
- [ ] 4.6 Add keyboard shortcuts for common navigation actions
- [ ] 4.7 Implement smooth interaction performance optimizations (<100ms response)
- [ ] 4.8 Add error handling for interaction edge cases
- [ ] 5.0 Develop Side Panel with Hover Information and CVD Reset Functionality
- [ ] 5.1 Create side panel layout using dash-bootstrap-components
- [ ] 5.2 Implement hover information display for OHLC data (timestamp, OHLC values, spread)
- [ ] 5.3 Implement hover information display for Volume data (timestamp, volume, buy/sell breakdown)
- [ ] 5.4 Implement hover information display for OBI data (timestamp, OBI value, bid/ask volumes)
- [ ] 5.5 Implement hover information display for CVD data (timestamp, CVD value, volume delta)
- [ ] 5.6 Add CVD reset functionality with click-to-reset on CVD chart
- [ ] 5.7 Implement visual markers for CVD reset points
- [ ] 5.8 Add CVD recalculation logic from reset point forward
- [ ] 5.9 Create control buttons in side panel (Reset CVD, Zoom Reset, etc.)
- [ ] 5.10 Optimize hover information update performance (<50ms latency)

66
tasks/tasks-prd-obi-cvd-metrics.md
View File

@@ -1,66 +0,0 @@
# Tasks: OBI and CVD Metrics Integration
Based on the PRD for integrating Order Book Imbalance (OBI) and Cumulative Volume Delta (CVD) calculations into the orderflow backtest system.
## Relevant Files
- `repositories/sqlite_repository.py` - Extend to support metrics table operations and batch insertions
- `repositories/test_metrics_repository.py` - Unit tests for metrics repository functionality
- `models.py` - Add new data models for metrics and update Book class
- `tests/test_models_metrics.py` - Unit tests for new metric models
- `storage.py` - Modify to integrate metric calculations during snapshot processing
- `tests/test_storage_metrics.py` - Unit tests for storage metric integration
- `strategies.py` - Enhance DefaultStrategy to calculate OBI and CVD metrics
- `tests/test_strategies_metrics.py` - Unit tests for strategy metric calculations
- `visualizer.py` - Extend to plot OBI and CVD curves beneath volume graphs
- `tests/test_visualizer_metrics.py` - Unit tests for metric visualization
- `parsers/metric_calculator.py` - New utility class for OBI and CVD calculations
- `tests/test_metric_calculator.py` - Unit tests for metric calculation logic
### Notes
- Unit tests should be placed alongside the code files they are testing
- Use `uv run pytest [optional/path/to/test/file]` to run tests following project standards
- Database schema changes require migration considerations for existing databases
## Tasks
- [ ] 1.0 Database Schema and Repository Updates
- [ ] 1.1 Create metrics table schema with proper indexes and foreign key constraints
- [ ] 1.2 Add metrics table creation method to SQLiteOrderflowRepository
- [ ] 1.3 Implement metrics insertion methods with batch support for performance
- [ ] 1.4 Add metrics querying methods (by timestamp range, snapshot_id)
- [ ] 1.5 Create database migration utility to add metrics table to existing databases
- [ ] 1.6 Add proper error handling and transaction management for metrics operations
- [ ] 2.0 Metric Calculation Engine
- [ ] 2.1 Create MetricCalculator class with OBI calculation method
- [ ] 2.2 Implement CVD calculation with incremental support and reset functionality
- [ ] 2.3 Add volume delta calculation for individual timestamps
- [ ] 2.4 Implement best bid/ask extraction from orderbook snapshots
- [ ] 2.5 Add edge case handling (empty orderbook, no trades, zero volume)
- [ ] 2.6 Create validation methods to ensure OBI values are within [-1, 1] range
- [ ] 3.0 Storage System Integration
- [ ] 3.1 Modify Storage.build_booktick_from_db to integrate metric calculations
- [ ] 3.2 Update _create_snapshots_from_rows to calculate and store metrics per snapshot
- [ ] 3.3 Implement memory optimization by removing full snapshot retention
- [ ] 3.4 Add metric persistence during snapshot processing
- [ ] 3.5 Update Book model to store only essential data (metrics + best bid/ask)
- [ ] 3.6 Add progress reporting for metric calculation during processing
- [ ] 4.0 Strategy Enhancement
- [ ] 4.1 Update DefaultStrategy to use MetricCalculator for OBI and CVD
- [ ] 4.2 Modify compute_OBI method to work with new metric calculation system
- [ ] 4.3 Add CVD computation method to DefaultStrategy
- [ ] 4.4 Return time-series data structures compatible with visualizer
- [ ] 4.5 Integrate metric calculation into on_booktick workflow
- [ ] 4.6 Add configuration options for CVD reset points and calculation parameters
- [ ] 5.0 Visualization Implementation
- [ ] 5.1 Extend Visualizer to load metrics data from database
- [ ] 5.2 Add OBI and CVD plotting methods beneath volume graphs
- [ ] 5.3 Implement shared X-axis time alignment across all charts (OHLC, volume, OBI, CVD)
- [ ] 5.4 Add 6-hour bar aggregation support for metrics visualization
- [ ] 5.5 Implement standard line styling for OBI and CVD curves
- [ ] 5.6 Make time resolution configurable for future flexibility