orderflow_backtest/tasks/prd-pyside6-pyqtgraph-migration.md

# Product Requirements Document: Migration from Dash/Plotly to PySide6/PyQtGraph

## Introduction/Overview

This PRD outlines the complete migration of the orderflow backtest visualization system from the current Dash/Plotly web-based implementation to a native desktop application using PySide6 and PyQtGraph. The migration addresses critical issues with the current implementation including async problems, debugging difficulties, performance bottlenecks, and data handling inefficiencies.

The goal is to create a robust, high-performance desktop application that provides better control over the codebase, eliminates current visualization bugs (particularly the CVD graph display issue), and enables future real-time trading strategy monitoring capabilities.

## Goals

1. **Eliminate Current Technical Issues**
   - Resolve async-related problems causing visualization failures
   - Fix CVD graph display issues that persist despite correct-looking code
   - Enable proper debugging capabilities with breakpoint support
   - Improve overall application performance and responsiveness

2. **Improve Development Experience**
   - Gain better control over the codebase through native Python implementation
   - Reduce dependency on intermediate file-based data exchange
   - Simplify the development and debugging workflow
   - Establish a foundation for future real-time capabilities

3. **Maintain and Enhance Visualization Capabilities**
   - Preserve all existing chart types and interactions
   - Improve performance for granular dataset handling
   - Prepare infrastructure for real-time data streaming
   - Enhance user experience through native desktop interface

## User Stories

1. **As a trading strategy developer**, I want to visualize OHLC data with volume, OBI, and CVD indicators in a single, synchronized view so that I can analyze market behavior patterns effectively.

2. **As a data analyst**, I want to zoom, pan, and select specific time ranges on charts so that I can focus on relevant market periods for detailed analysis.

3. **As a system developer**, I want to debug visualization issues with breakpoints and proper debugging tools so that I can identify and fix problems efficiently.

4. **As a performance-conscious user**, I want smooth chart rendering and interactions even with large, granular datasets so that my analysis workflow is not interrupted by lag or freezing.

5. **As a future trading system operator**, I want a foundation that can handle real-time data updates so that I can monitor live trading strategies effectively.

## Functional Requirements

### Core Visualization Components

1. **Main Chart Window**
   - The system must display OHLC candlestick charts in a primary plot area
   - The system must allow customizable time window selection for OHLC display
   - The system must synchronize all chart components to the same time axis

2. **Integrated Indicator Charts**
   - The system must display Volume bars below the OHLC chart
   - The system must display Order Book Imbalance (OBI) indicator
   - The system must display Cumulative Volume Delta (CVD) indicator
   - All indicators must share the same X-axis as the OHLC chart

3. **Depth Chart Visualization**
   - The system must display order book depth at selected time snapshots
   - The system must update depth visualization based on time selection
   - The system must provide clear bid/ask visualization

### User Interaction Features

4. **Chart Navigation**
   - The system must support zoom in/out functionality across all charts
   - The system must allow panning across time ranges
   - The system must provide time range selection capabilities
   - The system must support rectangle selection for detailed analysis

5. **Data Inspection**
   - The system must display mouseover information for all chart elements
   - The system must show precise values for OHLC, volume, OBI, and CVD data points
   - The system must provide crosshair functionality for precise data reading

### Technical Architecture

6. **Application Framework**
   - The system must be built using PySide6 for the GUI framework
   - The system must use PyQtGraph for all chart rendering and interactions
   - The system must implement a native desktop application architecture

7. **Data Integration**
   - The system must integrate with existing data processing modules (metrics_calculator, ohlc_processor, orderbook_manager)
   - The system must eliminate dependency on intermediate JSON files for data display
   - The system must support direct in-memory data transfer between processing and visualization

8. **Performance Requirements**
   - The system must handle granular datasets efficiently without UI blocking
   - The system must provide smooth chart interactions (zoom, pan, selection)
   - The system must render updates in less than 100ms for typical dataset sizes

### Development and Debugging

9. **Code Quality**
   - The system must be fully debuggable with standard Python debugging tools
   - The system must follow the existing project architecture patterns
   - The system must maintain clean separation between data processing and visualization

## Non-Goals (Out of Scope)

1. **Web Interface Maintenance** - The existing Dash/Plotly implementation will be completely replaced, not maintained in parallel

2. **Backward Compatibility** - No requirement to maintain compatibility with existing Dash/Plotly components or web-based deployment

3. **Multi-Platform Distribution** - Initial focus on development environment only, not packaging for distribution

4. **Real-Time Implementation** - While the architecture should support future real-time capabilities, the initial migration will focus on historical data visualization

5. **Advanced Chart Types** - Only migrate existing chart types; new visualization features are out of scope for this migration

## Design Considerations

### User Interface Layout
- **Main Window Structure**: Primary chart area with integrated indicators below
- **Control Panel**: Side panel or toolbar for time range selection and chart configuration
- **Status Bar**: Display current data range, loading status, and performance metrics
- **Menu System**: File operations, view options, and application settings

### PyQtGraph Integration
- **Plot Organization**: Use PyQtGraph's PlotWidget for main charts with linked axes
- **Custom Plot Items**: Implement custom plot items for OHLC candlesticks and depth visualization
- **Performance Optimization**: Utilize PyQtGraph's fast plotting capabilities for large datasets

### Data Flow Architecture
- **Direct Memory Access**: Replace JSON file intermediates with direct Python object passing
- **Lazy Loading**: Implement efficient data loading strategies for large time ranges
- **Caching Strategy**: Cache processed data to improve navigation performance

## Technical Considerations

### Dependencies and Integration
- **PySide6**: Main GUI framework, provides native desktop capabilities
- **PyQtGraph**: High-performance plotting library, optimized for real-time data
- **Existing Modules**: Maintain integration with metrics_calculator.py, ohlc_processor.py, orderbook_manager.py
- **Database Integration**: Continue using existing SQLite database through db_interpreter.py

### Migration Strategy (Iterative Implementation)
- **Phase 1**: Basic PySide6 window with single PyQtGraph plot
- **Phase 2**: OHLC candlestick chart implementation
- **Phase 3**: Volume, OBI, and CVD indicator integration
- **Phase 4**: Depth chart implementation
- **Phase 5**: User interaction features (zoom, pan, selection)
- **Phase 6**: Data integration and performance optimization

### Performance Considerations
- **Memory Management**: Efficient data structure handling for large datasets
- **Rendering Optimization**: Use PyQtGraph's ViewBox and plotting optimizations
- **Thread Safety**: Proper handling of data processing in background threads
- **Resource Cleanup**: Proper cleanup of chart objects and data structures

## Success Metrics

### Technical Success Criteria
1. **Bug Resolution**: CVD graph displays correctly and all existing visualization bugs are resolved
2. **Performance Improvement**: Chart interactions respond within 100ms for typical datasets
3. **Debugging Capability**: Developers can set breakpoints and debug visualization code effectively
4. **Data Handling**: Elimination of intermediate JSON files reduces data transfer overhead by 50%

### User Experience Success Criteria
1. **Feature Parity**: All existing chart types and interactions are preserved and functional
2. **Responsiveness**: Application feels more responsive than the current Dash implementation
3. **Stability**: No crashes or freezing during normal chart operations
4. **Visual Quality**: Charts render clearly with proper scaling and anti-aliasing

### Development Success Criteria
1. **Code Maintainability**: New codebase follows established project patterns and is easier to maintain
2. **Development Velocity**: Future visualization features can be implemented more quickly
3. **Testing Capability**: Comprehensive testing can be performed with proper debugging tools
4. **Architecture Foundation**: System is ready for future real-time data integration

## Open Questions

1. **Data Loading Strategy**: Should we implement progressive loading for very large datasets, or rely on existing data chunking mechanisms?

2. **Configuration Management**: How should chart configuration and user preferences be stored and managed in the desktop application?

3. **Error Handling**: What specific error handling and user feedback mechanisms should be implemented for data loading and processing failures?

4. **Performance Monitoring**: Should we include built-in performance monitoring and profiling tools in the application?

5. **Future Real-Time Integration**: What specific interface patterns should be established now to facilitate future real-time data streaming integration?

## Implementation Approach

This migration will follow the iterative development workflow with explicit approval checkpoints between each phase. Each implementation phase will be:
- Limited to manageable scope (≤250 lines per module)
- Tested immediately after implementation
- Integrated with existing data processing modules
- Validated for performance and functionality before proceeding to the next phase

The implementation will begin with basic PySide6 application structure and progressively add PyQtGraph visualization capabilities while maintaining integration with the existing data processing pipeline.