review project state and update tasks and context

CONTEXT.md

@@ -0,0 +1,92 @@
+# Project Context: Simplified Crypto Trading Bot Platform
+
+This document provides a comprehensive overview of the project's architecture, technology stack, conventions, and current implementation status, following the guidelines in `context-management.md`.
+
+## 1. Architecture Overview
+
+The platform is a **monolithic application** built with Python, designed for rapid development and internal testing of crypto trading strategies. The architecture is modular, with clear separation between components to facilitate future migration to microservices if needed.
+
+### Core Components
+- **Data Collection Service**: A standalone, multi-process service responsible for collecting real-time market data from exchanges (currently OKX). It uses a robust `BaseDataCollector` abstraction and specific exchange implementations (e.g., `OKXCollector`). Data is processed, aggregated into OHLCV candles, and stored.
+- **Database**: PostgreSQL with the TimescaleDB extension (though currently using a "clean" schema without hypertables for simplicity). It stores market data, bot configurations, trading signals, and performance metrics. SQLAlchemy is used as the ORM.
+- **Real-time Messaging**: Redis is used for pub/sub messaging, intended for real-time data distribution between components (though its use in the dashboard is currently deferred).
+- **Dashboard & API**: A Dash application serves as the main user interface for visualization, bot management, and system monitoring. The underlying Flask server can be extended for REST APIs.
+- **Strategy Engine & Bot Manager**: (Not yet implemented) This component will be responsible for executing trading logic, managing bot lifecycles, and tracking virtual portfolios.
+- **Backtesting Engine**: (Not yet implemented) This will provide capabilities to test strategies against historical data.
+
+### Data Flow
+1.  The `DataCollectionService` connects to the OKX WebSocket API.
+2.  Raw trade data is received and processed by `OKXDataProcessor`.
+3.  Trades are aggregated into OHLCV candles (1m, 5m, etc.).
+4.  Both raw trade data and processed OHLCV candles are stored in the PostgreSQL database.
+5.  (Future) The Strategy Engine will consume OHLCV data to generate trading signals.
+6.  The Dashboard reads data from the database to provide visualizations and system health monitoring.
+
+## 2. Technology Stack
+
+- **Backend**: Python 3.10+
+- **Web Framework**: Dash with Dash Bootstrap Components for the frontend UI.
+- **Database**: PostgreSQL 14+. SQLAlchemy for ORM. Alembic for migrations.
+- **Messaging**: Redis for pub/sub.
+- **Data & Numerics**: pandas for data manipulation (especially in backtesting).
+- **Package Management**: `uv`
+- **Containerization**: Docker and Docker Compose for setting up the development environment (PostgreSQL, Redis, etc.).
+
+## 3. Coding Conventions
+
+- **Modular Design**: Code is organized into modules with a clear purpose (e.g., `data`, `database`, `dashboard`). See `architecture.md` for more details.
+- **Naming Conventions**:
+    - **Classes**: `PascalCase` (e.g., `MarketData`, `BaseDataCollector`).
+    - **Functions & Methods**: `snake_case` (e.g., `get_system_health_layout`, `connect`).
+    - **Variables & Attributes**: `snake_case` (e.g., `exchange_name`, `_ws_client`).
+    - **Constants**: `UPPER_SNAKE_CASE` (e.g., `MAX_RECONNECT_ATTEMPTS`).
+    - **Modules**: `snake_case.py` (e.g., `collector_manager.py`).
+    - **Private Attributes/Methods**: Use a single leading underscore `_` (e.g., `_process_message`). Avoid double underscores unless for name mangling in classes.
+- **File Organization & Code Structure**:
+    - **Directory Structure**: Top-level directories separate major concerns (`data`, `database`, `dashboard`, `strategies`). Sub-packages should be used for further organization (e.g., `data/exchanges/okx`).
+    - **Module Structure**: Within a Python module (`.py` file), the preferred order is:
+        1.  Module-level docstring explaining its purpose.
+        2.  Imports (see pattern below).
+        3.  Module-level constants (`ALL_CAPS`).
+        4.  Custom exception classes.
+        5.  Data classes or simple data structures.
+        6.  Helper functions (if any, typically private `_helper()`).
+        7.  Main business logic classes.
+    - **`__init__.py`**: Use `__init__.py` files to define a package's public API and simplify imports for consumers of the package.
+- **Import/Export Patterns**:
+    - **Grouping**: Imports should be grouped in the following order, with a blank line between each group:
+        1.  Standard library imports (e.g., `asyncio`, `datetime`).
+        2.  Third-party library imports (e.g., `dash`, `sqlalchemy`).
+        3.  Local application imports (e.g., `from utils.logger import get_logger`).
+    - **Style**: Use absolute imports (`from data.base_collector import ...`) over relative imports (`from ..base_collector import ...`) for better readability and to avoid ambiguity.
+    - **Exports**: To create a clean public API for a package, import the desired classes/functions into the `__init__.py` file. This allows users to import directly from the package (e.g., `from data.exchanges import ExchangeFactory`) instead of from the specific submodule.
+- **Abstract Base Classes**: Used to define common interfaces, as seen in `data/base_collector.py`.
+- **Configuration**: Bot and strategy parameters are managed via JSON files in `config/`. Centralized application settings are handled by `config/settings.py`.
+- **Logging**: A unified logging system is available in `utils/logger.py` and should be used across all components for consistent output.
+- **Type Hinting**: Mandatory for all function signatures (parameters and return values) for clarity and static analysis.
+- **Error Handling**: Custom, specific exceptions should be defined (e.g., `DataCollectorError`). Use `try...except` blocks to handle potential failures gracefully and provide informative error messages.
+- **Database Access**: A `DatabaseManager` in `database/connection.py` provides a centralized way to handle database sessions and connections. All database operations should ideally go through an operations/repository layer.
+
+## 4. Current Implementation Status
+
+### Completed Features
+- **Database Foundation**: The database schema is fully defined in `database/models.py` and `database/schema_clean.sql`, with all necessary tables, indexes, and relationships. Database connection management is robust.
+- **Data Collection System**: A highly robust and asynchronous data collection service is in place. It supports OKX, handles WebSocket connections, processes data, aggregates OHLCV candles, and stores data reliably. It features health monitoring and automatic restarts.
+- **Basic Dashboard**: A functional dashboard exists.
+  - **System Health Monitoring**: A comprehensive page shows the real-time status of the data collection service, database, Redis, and system performance (CPU/memory).
+  - **Data Visualization**: Price charts with technical indicator overlays are implemented.
+
+### Work in Progress / To-Do
+The core business logic of the application is yet to be implemented. The main remaining tasks are:
+- **Strategy Engine and Bot Management (Task Group 4.0)**:
+  - Designing the base strategy interface.
+  - Implementing bot lifecycle management (create, run, stop).
+  - Signal generation and virtual portfolio tracking.
+- **Advanced Dashboard Features (Task Group 5.0)**:
+  - Building the UI for managing bots and configuring strategies.
+- **Backtesting Engine (Task Group 6.0)**:
+  - Implementing the engine to test strategies on historical data.
+- **Real-Time Trading Simulation (Task Group 7.0)**:
+  - Executing virtual trades based on signals.
+
+The project has a solid foundation. The next phase of development should focus on implementing the trading logic and user-facing bot management features. 

tasks/PRD-tasks.md

@@ -1,5 +1,6 @@
 ## Relevant Files
 
+- `CONTEXT.md` - **NEW**: Project overview, architecture, tech stack, and implementation status.
 - `app.py` - Main Dash application entry point and dashboard interface
 - `bot_manager.py` - Bot lifecycle management and coordination
 - `database/models.py` - PostgreSQL database models and schema definitions (updated to match schema_clean.sql)
@@ -14,7 +15,7 @@
 - `data/collector_manager.py` - Centralized collector management with health monitoring, auto-recovery, and coordinated lifecycle management
 - `data/collection_service.py` - Production-ready data collection service with clean logging, multi-exchange support, and robust error handling
 - `data/__init__.py` - Data collection package initialization
-- `data/okx_collector.py` - OKX API integration for real-time market data collection
+- `data/exchanges/okx/collector.py` - OKX API integration for real-time market data collection (Corrected Path)
 - `data/aggregator.py` - OHLCV candle aggregation and processing
 - `data/common/indicators.py` - Technical indicators module with SMA, EMA, RSI, MACD, and Bollinger Bands calculations optimized for sparse OHLCV data
 - `strategies/base_strategy.py` - Base strategy class and interface
@@ -90,65 +91,59 @@
   - [x] 3.9 Add data export functionality for analysis (CSV/JSON export)
   - [x] 3.10 Unit test basic dashboard components and data visualization
 
-- [ ] 4.0 Strategy Engine and Bot Management Framework
-  - [ ] 4.1 Design and implement base strategy interface class
-  - [ ] 4.2 Create EMA crossover strategy as reference implementation
-  - [ ] 4.3 Implement JSON-based strategy parameter configuration system
-  - [ ] 4.4 Build bot lifecycle management (create, start, stop, pause, delete)
-  - [ ] 4.5 Create signal generation and processing logic
-  - [ ] 4.6 Implement virtual portfolio management and balance tracking
-  - [ ] 4.7 Add bot status monitoring and heartbeat system
-  - [ ] 4.8 Create bot configuration management with JSON files
-  - [ ] 4.9 Implement multi-bot coordination and resource management
-  - [ ] 4.10 Unit test strategy engine and bot management functionality
+- [ ] 4.0 Strategy Engine Foundation
+  - [ ] 4.1 Design and implement `BaseStrategy` abstract class in `strategies/base_strategy.py` with `process_data` and `get_indicators` methods.
+  - [ ] 4.2 Implement `EMA Crossover` strategy in `strategies/ema_crossover.py`, inheriting from `BaseStrategy`.
+  - [ ] 4.3 Implement `MACD` strategy in `strategies/macd_strategy.py` to provide another reference implementation.
+  - [ ] 4.4 Implement `RSI` strategy in `strategies/rsi_strategy.py` for momentum-based signals.
+  - [ ] 4.5 Create a strategy factory or registry in `strategies/factory.py` to dynamically load strategies from their configuration files.
+  - [ ] 4.6 Implement a JSON-based parameter configuration system in `config/strategies/` for each strategy type.
+  - [ ] 4.7 Create comprehensive unit tests in `tests/test_strategies.py` to validate the signal generation logic for each strategy under various market conditions.
 
-- [ ] 5.0 Advanced Dashboard Features and Bot Interface
-  - [ ] 5.1 Build bot management interface (start/stop controls, status indicators)
-  - [ ] 5.2 Create bot configuration forms for JSON parameter editing
-  - [ ] 5.3 Add strategy signal overlay on price charts
-  - [ ] 5.4 Implement bot status monitoring dashboard
-  - [ ] 5.5 Create system health and performance monitoring interface
-  - [ ] 5.6 Unit test advanced dashboard features and bot interface
+- [ ] 5.0 Vectorized Backtesting Engine
+  - [ ] 5.1 Design `BacktestingEngine` class in `backtesting/engine.py` to orchestrate the backtesting process.
+  - [ ] 5.2 Implement historical data loading from the database using the existing `MarketDataRepository`.
+  - [ ] 5.3 Implement the core vectorized backtesting loop using pandas for efficient signal and portfolio calculation.
+  - [ ] 5.4 Integrate the strategy factory to run tests on any registered strategy.
+  - [ ] 5.5 Create `backtesting/performance.py` to calculate key metrics (Sharpe Ratio, Max Drawdown, Win Rate, Total Return).
+  - [ ] 5.6 Implement realistic fee modeling (e.g., 0.1% per trade) and slippage simulation.
+  - [ ] 5.7 Define a standardized `BacktestResult` data structure to store trade history, portfolio progression, and final metrics.
+  - [ ] 5.8 Create unit tests in `tests/test_backtesting.py` to verify engine calculations and performance metrics against known outcomes.
 
-- [ ] 6.0 Backtesting Engine and Performance Analytics
-  - [ ] 6.1 Implement historical data loading from database or file
-  - [ ] 6.2 Create vectorized backtesting engine using pandas operations
-  - [ ] 6.3 Build performance metrics calculation (Sharpe ratio, drawdown, win rate, total return)
-  - [ ] 6.4 Implement realistic fee modeling (0.1% per trade for OKX)
-  - [ ] 6.5 Add look-ahead bias prevention with proper timestamp handling
-  - [ ] 6.6 Create parallel backtesting system for multiple strategies
-  - [ ] 6.7 Create strategy comparison and reporting functionality
-  - [ ] 6.8 Build backtesting results visualization and export
-  - [ ] 6.9 Implement configurable test periods (1 day to 24 months)
-  - [ ] 6.10 Unit test backtesting engine and performance analytics
+- [ ] 6.0 Bot Management & Real-Time Simulation Engine
+  - [ ] 6.1 Design `BotManager` class in `bot/manager.py` to handle the lifecycle (create, start, stop, monitor) of multiple bot instances.
+  - [ ] 6.2 Create a `Bot` class in `bot/instance.py` to encapsulate the state of a single trading bot (config, portfolio, status).
+  - [ ] 6.3 Implement a `VirtualPortfolio` class in `bot/portfolio.py` to track virtual assets, balances, and P&L.
+  - [ ] 6.4 Develop a simulation loop that processes new market data (initially from the database, mimicking real-time) and triggers strategies.
+  - [ ] 6.5 Implement the simulated trade execution logic, updating the `VirtualPortfolio` and recording trades in the database.
+  - [ ] 6.6 Implement a heartbeat system where each active bot updates its `last_heartbeat` in the `bots` table.
+  - [ ] 6.7 Create a monitoring process within the `BotManager` to check for stalled or crashed bots.
+  - [ ] 6.8 Create unit tests in `tests/test_bot_management.py` for bot state transitions, portfolio updates, and trade execution logic.
 
-- [ ] 7.0 Real-Time Trading Simulation
-  - [ ] 7.1 Implement virtual trading execution engine
-  - [ ] 7.2 Create order management system (market, limit orders)
-  - [ ] 7.3 Build trade execution logic with proper timing
-  - [ ] 7.4 Implement position tracking and balance updates
-  - [ ] 7.5 Add risk management controls (stop-loss, take-profit, position sizing)
-  - [ ] 7.6 Create trade reconciliation and confirmation system
-  - [ ] 7.7 Implement fee calculation and tracking
-  - [ ] 7.8 Add emergency stop mechanisms for bots
-  - [ ] 7.9 Unit test real-time trading simulation
+- [ ] 7.0 Dashboard Integration for Trading Operations
+  - [ ] 7.1 Create a new dashboard layout in `dashboard/layouts/trading.py` for bot management and backtesting.
+  - [ ] 7.2 Build a bot creation form using Dash Bootstrap Components to select a symbol, strategy, and configuration file.
+  - [ ] 7.3 Implement callbacks in `dashboard/callbacks/trading.py` to save new bot configurations to the database.
+  - [ ] 7.4 Create a table of all bots from the database, showing their status with "Start/Stop" control buttons.
+  - [ ] 7.5 Implement callbacks to trigger the `BotManager` to start and stop bots based on user interaction.
+  - [ ] 7.6 Design a simple UI for initiating backtests by selecting a strategy, symbol, and date range.
+  - [ ] 7.7 Implement a callback to run the `BacktestingEngine` in a separate process/thread to avoid blocking the UI.
 
 - [ ] 8.0 Portfolio Visualization and Trade Analytics
-  - [ ] 8.1 Build portfolio performance visualization charts (equity curve, drawdown, win rate)
-  - [ ] 8.2 Create trade history table with P&L calculations
-  - [ ] 8.3 Implement real-time portfolio tracking and updates
-  - [ ] 8.4 Add performance comparison charts between multiple bots
-  - [ ] 8.5 Create trade analytics and statistics dashboard
-  - [ ] 8.6 Unit test portfolio visualization and trade analytics
+  - [ ] 8.1 Create a new layout in `dashboard/layouts/performance.py` for displaying backtest and bot performance results.
+  - [ ] 8.2 Implement an interactive equity curve chart with Plotly to visualize portfolio value over time.
+  - [ ] 8.3 Display key performance metrics (Sharpe Ratio, Drawdown, etc.) in `dbc.Card` components.
+  - [ ] 8.4 Add a `dash_ag_grid` or `DataTable` to show detailed trade history.
+  - [ ] 8.5 Enhance the main price chart to overlay buy/sell signals from a selected backtest or running bot.
+  - [ ] 8.6 Implement callbacks in `dashboard/callbacks/performance.py` to fetch and display results from a completed backtest or an active bot.
 
-- [ ] 9.0 Documentation and User Guide
-  - [ ] 9.1 Write comprehensive README with setup instructions
-  - [ ] 9.2 Create API documentation for all modules
-  - [ ] 9.3 Document strategy development guidelines
-  - [ ] 9.4 Write user guide for bot configuration and management
-  - [ ] 9.5 Create troubleshooting guide for common issues
-  - [ ] 9.6 Document database schema and data flow
-  - [ ] 9.7 Add code comments and docstrings throughout codebase
+- [ ] 9.0 System Finalization and Documentation
+  - [ ] 9.1 Write comprehensive documentation in `/docs/guides/` for strategy development and bot configuration.
+  - [ ] 9.2 Add detailed docstrings and code comments to all new classes and complex functions.
+  - [ ] 9.3 Perform end-to-end integration testing with 5+ bots running concurrently for 24+ hours.
+  - [ ] 9.4 Implement comprehensive input validation and error handling on all dashboard components.
+  - [ ] 9.5 Create a final deployment checklist and update the main `README.md` with usage instructions.
+  - [ ] 9.6 Review and clean up the entire codebase, ensuring consistency with the conventions in `CONTEXT.md`.
 
 - [ ] 10.0 Deployment and Monitoring Setup
   - [ ] 10.1 Create Docker containers for all services