review project state and update tasks and context

tasks/PRD-tasks.md

@@ -1,5 +1,6 @@
 ## Relevant Files
 
+- `CONTEXT.md` - **NEW**: Project overview, architecture, tech stack, and implementation status.
 - `app.py` - Main Dash application entry point and dashboard interface
 - `bot_manager.py` - Bot lifecycle management and coordination
 - `database/models.py` - PostgreSQL database models and schema definitions (updated to match schema_clean.sql)
@@ -14,7 +15,7 @@
 - `data/collector_manager.py` - Centralized collector management with health monitoring, auto-recovery, and coordinated lifecycle management
 - `data/collection_service.py` - Production-ready data collection service with clean logging, multi-exchange support, and robust error handling
 - `data/__init__.py` - Data collection package initialization
-- `data/okx_collector.py` - OKX API integration for real-time market data collection
+- `data/exchanges/okx/collector.py` - OKX API integration for real-time market data collection (Corrected Path)
 - `data/aggregator.py` - OHLCV candle aggregation and processing
 - `data/common/indicators.py` - Technical indicators module with SMA, EMA, RSI, MACD, and Bollinger Bands calculations optimized for sparse OHLCV data
 - `strategies/base_strategy.py` - Base strategy class and interface
@@ -90,65 +91,59 @@
   - [x] 3.9 Add data export functionality for analysis (CSV/JSON export)
   - [x] 3.10 Unit test basic dashboard components and data visualization
 
-- [ ] 4.0 Strategy Engine and Bot Management Framework
-  - [ ] 4.1 Design and implement base strategy interface class
-  - [ ] 4.2 Create EMA crossover strategy as reference implementation
-  - [ ] 4.3 Implement JSON-based strategy parameter configuration system
-  - [ ] 4.4 Build bot lifecycle management (create, start, stop, pause, delete)
-  - [ ] 4.5 Create signal generation and processing logic
-  - [ ] 4.6 Implement virtual portfolio management and balance tracking
-  - [ ] 4.7 Add bot status monitoring and heartbeat system
-  - [ ] 4.8 Create bot configuration management with JSON files
-  - [ ] 4.9 Implement multi-bot coordination and resource management
-  - [ ] 4.10 Unit test strategy engine and bot management functionality
+- [ ] 4.0 Strategy Engine Foundation
+  - [ ] 4.1 Design and implement `BaseStrategy` abstract class in `strategies/base_strategy.py` with `process_data` and `get_indicators` methods.
+  - [ ] 4.2 Implement `EMA Crossover` strategy in `strategies/ema_crossover.py`, inheriting from `BaseStrategy`.
+  - [ ] 4.3 Implement `MACD` strategy in `strategies/macd_strategy.py` to provide another reference implementation.
+  - [ ] 4.4 Implement `RSI` strategy in `strategies/rsi_strategy.py` for momentum-based signals.
+  - [ ] 4.5 Create a strategy factory or registry in `strategies/factory.py` to dynamically load strategies from their configuration files.
+  - [ ] 4.6 Implement a JSON-based parameter configuration system in `config/strategies/` for each strategy type.
+  - [ ] 4.7 Create comprehensive unit tests in `tests/test_strategies.py` to validate the signal generation logic for each strategy under various market conditions.
 
-- [ ] 5.0 Advanced Dashboard Features and Bot Interface
-  - [ ] 5.1 Build bot management interface (start/stop controls, status indicators)
-  - [ ] 5.2 Create bot configuration forms for JSON parameter editing
-  - [ ] 5.3 Add strategy signal overlay on price charts
-  - [ ] 5.4 Implement bot status monitoring dashboard
-  - [ ] 5.5 Create system health and performance monitoring interface
-  - [ ] 5.6 Unit test advanced dashboard features and bot interface
+- [ ] 5.0 Vectorized Backtesting Engine
+  - [ ] 5.1 Design `BacktestingEngine` class in `backtesting/engine.py` to orchestrate the backtesting process.
+  - [ ] 5.2 Implement historical data loading from the database using the existing `MarketDataRepository`.
+  - [ ] 5.3 Implement the core vectorized backtesting loop using pandas for efficient signal and portfolio calculation.
+  - [ ] 5.4 Integrate the strategy factory to run tests on any registered strategy.
+  - [ ] 5.5 Create `backtesting/performance.py` to calculate key metrics (Sharpe Ratio, Max Drawdown, Win Rate, Total Return).
+  - [ ] 5.6 Implement realistic fee modeling (e.g., 0.1% per trade) and slippage simulation.
+  - [ ] 5.7 Define a standardized `BacktestResult` data structure to store trade history, portfolio progression, and final metrics.
+  - [ ] 5.8 Create unit tests in `tests/test_backtesting.py` to verify engine calculations and performance metrics against known outcomes.
 
-- [ ] 6.0 Backtesting Engine and Performance Analytics
-  - [ ] 6.1 Implement historical data loading from database or file
-  - [ ] 6.2 Create vectorized backtesting engine using pandas operations
-  - [ ] 6.3 Build performance metrics calculation (Sharpe ratio, drawdown, win rate, total return)
-  - [ ] 6.4 Implement realistic fee modeling (0.1% per trade for OKX)
-  - [ ] 6.5 Add look-ahead bias prevention with proper timestamp handling
-  - [ ] 6.6 Create parallel backtesting system for multiple strategies
-  - [ ] 6.7 Create strategy comparison and reporting functionality
-  - [ ] 6.8 Build backtesting results visualization and export
-  - [ ] 6.9 Implement configurable test periods (1 day to 24 months)
-  - [ ] 6.10 Unit test backtesting engine and performance analytics
+- [ ] 6.0 Bot Management & Real-Time Simulation Engine
+  - [ ] 6.1 Design `BotManager` class in `bot/manager.py` to handle the lifecycle (create, start, stop, monitor) of multiple bot instances.
+  - [ ] 6.2 Create a `Bot` class in `bot/instance.py` to encapsulate the state of a single trading bot (config, portfolio, status).
+  - [ ] 6.3 Implement a `VirtualPortfolio` class in `bot/portfolio.py` to track virtual assets, balances, and P&L.
+  - [ ] 6.4 Develop a simulation loop that processes new market data (initially from the database, mimicking real-time) and triggers strategies.
+  - [ ] 6.5 Implement the simulated trade execution logic, updating the `VirtualPortfolio` and recording trades in the database.
+  - [ ] 6.6 Implement a heartbeat system where each active bot updates its `last_heartbeat` in the `bots` table.
+  - [ ] 6.7 Create a monitoring process within the `BotManager` to check for stalled or crashed bots.
+  - [ ] 6.8 Create unit tests in `tests/test_bot_management.py` for bot state transitions, portfolio updates, and trade execution logic.
 
-- [ ] 7.0 Real-Time Trading Simulation
-  - [ ] 7.1 Implement virtual trading execution engine
-  - [ ] 7.2 Create order management system (market, limit orders)
-  - [ ] 7.3 Build trade execution logic with proper timing
-  - [ ] 7.4 Implement position tracking and balance updates
-  - [ ] 7.5 Add risk management controls (stop-loss, take-profit, position sizing)
-  - [ ] 7.6 Create trade reconciliation and confirmation system
-  - [ ] 7.7 Implement fee calculation and tracking
-  - [ ] 7.8 Add emergency stop mechanisms for bots
-  - [ ] 7.9 Unit test real-time trading simulation
+- [ ] 7.0 Dashboard Integration for Trading Operations
+  - [ ] 7.1 Create a new dashboard layout in `dashboard/layouts/trading.py` for bot management and backtesting.
+  - [ ] 7.2 Build a bot creation form using Dash Bootstrap Components to select a symbol, strategy, and configuration file.
+  - [ ] 7.3 Implement callbacks in `dashboard/callbacks/trading.py` to save new bot configurations to the database.
+  - [ ] 7.4 Create a table of all bots from the database, showing their status with "Start/Stop" control buttons.
+  - [ ] 7.5 Implement callbacks to trigger the `BotManager` to start and stop bots based on user interaction.
+  - [ ] 7.6 Design a simple UI for initiating backtests by selecting a strategy, symbol, and date range.
+  - [ ] 7.7 Implement a callback to run the `BacktestingEngine` in a separate process/thread to avoid blocking the UI.
 
 - [ ] 8.0 Portfolio Visualization and Trade Analytics
-  - [ ] 8.1 Build portfolio performance visualization charts (equity curve, drawdown, win rate)
-  - [ ] 8.2 Create trade history table with P&L calculations
-  - [ ] 8.3 Implement real-time portfolio tracking and updates
-  - [ ] 8.4 Add performance comparison charts between multiple bots
-  - [ ] 8.5 Create trade analytics and statistics dashboard
-  - [ ] 8.6 Unit test portfolio visualization and trade analytics
+  - [ ] 8.1 Create a new layout in `dashboard/layouts/performance.py` for displaying backtest and bot performance results.
+  - [ ] 8.2 Implement an interactive equity curve chart with Plotly to visualize portfolio value over time.
+  - [ ] 8.3 Display key performance metrics (Sharpe Ratio, Drawdown, etc.) in `dbc.Card` components.
+  - [ ] 8.4 Add a `dash_ag_grid` or `DataTable` to show detailed trade history.
+  - [ ] 8.5 Enhance the main price chart to overlay buy/sell signals from a selected backtest or running bot.
+  - [ ] 8.6 Implement callbacks in `dashboard/callbacks/performance.py` to fetch and display results from a completed backtest or an active bot.
 
-- [ ] 9.0 Documentation and User Guide
-  - [ ] 9.1 Write comprehensive README with setup instructions
-  - [ ] 9.2 Create API documentation for all modules
-  - [ ] 9.3 Document strategy development guidelines
-  - [ ] 9.4 Write user guide for bot configuration and management
-  - [ ] 9.5 Create troubleshooting guide for common issues
-  - [ ] 9.6 Document database schema and data flow
-  - [ ] 9.7 Add code comments and docstrings throughout codebase
+- [ ] 9.0 System Finalization and Documentation
+  - [ ] 9.1 Write comprehensive documentation in `/docs/guides/` for strategy development and bot configuration.
+  - [ ] 9.2 Add detailed docstrings and code comments to all new classes and complex functions.
+  - [ ] 9.3 Perform end-to-end integration testing with 5+ bots running concurrently for 24+ hours.
+  - [ ] 9.4 Implement comprehensive input validation and error handling on all dashboard components.
+  - [ ] 9.5 Create a final deployment checklist and update the main `README.md` with usage instructions.
+  - [ ] 9.6 Review and clean up the entire codebase, ensuring consistency with the conventions in `CONTEXT.md`.
 
 - [ ] 10.0 Deployment and Monitoring Setup
   - [ ] 10.1 Create Docker containers for all services