TCPDashboard/tasks/MAIN-task-list.md

## Relevant Files

- `CONTEXT.md` - **NEW**: Project overview, architecture, tech stack, and implementation status.
- `app.py` - Main Dash application entry point and dashboard interface
- `bot_manager.py` - Bot lifecycle management and coordination
- `database/models.py` - PostgreSQL database models and schema definitions (updated to match schema_clean.sql)
- `database/schema_clean.sql` - Clean database schema without hypertables (actively used, includes raw_trades table)
- `database/schema.sql` - Complete database schema with TimescaleDB hypertables (for future optimization)
- `database/connection.py` - Database connection utility with connection pooling, session management, and raw data utilities
- `database/redis_manager.py` - Redis connection utility with pub/sub messaging for real-time data distribution
- `database/migrations/` - Alembic migration system for database schema versioning and updates
- `database/init/init.sql` - Docker initialization script for automatic database setup
- `database/init/schema_clean.sql` - Copy of clean schema for Docker initialization
- `data/base_collector.py` - Abstract base class for all data collectors with standardized interface, error handling, data validation, health monitoring, and auto-restart capabilities
- `data/collector_manager.py` - Centralized collector management with health monitoring, auto-recovery, and coordinated lifecycle management
- `data/collection_service.py` - Production-ready data collection service with clean logging, multi-exchange support, and robust error handling
- `data/__init__.py` - Data collection package initialization
- `data/exchanges/okx/collector.py` - OKX API integration for real-time market data collection (Corrected Path)
- `data/aggregator.py` - OHLCV candle aggregation and processing
- `data/common/indicators.py` - Technical indicators module with SMA, EMA, RSI, MACD, and Bollinger Bands calculations optimized for sparse OHLCV data
- `strategies/base_strategy.py` - Base strategy class and interface
- `strategies/ema_crossover.py` - Example EMA crossover strategy implementation
- `components/dashboard.py` - Dashboard UI components and layouts
- `components/charts.py` - Price charts and visualization components
- `backtesting/engine.py` - Backtesting engine for historical strategy testing
- `backtesting/performance.py` - Performance metrics calculation
- `config/bot_configs/` - Directory for JSON bot configuration files
- `config/strategies/` - Directory for JSON strategy parameter files
- `config/settings.py` - Centralized configuration settings using Pydantic
- `scripts/dev.py` - Development setup and management script
- `scripts/start_data_collection.py` - Simple script to start the data collection service with clean output
- `scripts/production_clean.py` - Clean production OKX data collector script (adapted for service development)
- `scripts/monitor_clean.py` - Clean database monitor for production data collection status
- `scripts/init_database.py` - Database initialization and verification script
- `scripts/test_models.py` - Test script for SQLAlchemy models integration verification
- `utils/logger.py` - Enhanced unified logging system with verbose console output, automatic cleanup, and configurable retention [USE THIS FOR ALL LOGGING]
- `alembic.ini` - Alembic configuration for database migrations
- `requirements.txt` - Python dependencies managed by UV
- `docker-compose.yml` - Docker services configuration with TimescaleDB support
- `tests/test_strategies.py` - Unit tests for strategy implementations
- `tests/test_bot_manager.py` - Unit tests for bot management functionality
- `tests/test_data_collection.py` - Unit tests for data collection and aggregation
- `tests/test_data_collection_service.py` - Comprehensive unit tests for the DataCollectionService (25 tests)
- `tests/test_base_collector.py` - Comprehensive unit tests for the BaseDataCollector abstract class (13 tests)
- `tests/test_collector_manager.py` - Comprehensive unit tests for the CollectorManager with health monitoring (14 tests)
- `tests/test_logging_enhanced.py` - Comprehensive unit tests for enhanced logging features (16 tests)
- `tests/test_indicators.py` - Comprehensive unit tests for technical indicators module (18 tests)
- `docs/setup.md` - Comprehensive setup guide for new machines and environments
- `docs/logging.md` - Complete documentation for the enhanced unified logging system
- `docs/data-collection-service.md` - Complete documentation for the data collection service with usage examples, configuration, and deployment guide
- `docs/components/technical-indicators.md` - Complete documentation for the technical indicators module with usage examples and integration guide
- `dashboard/layouts/system_health.py` - Enhanced system health monitoring layout with comprehensive market data monitoring using Mantine components
- `dashboard/callbacks/system_health.py` - Enhanced system health callbacks with real-time data collection monitoring, database statistics, Redis monitoring, and performance metrics using Mantine components

## Tasks

- [ ] 1.0 Database Foundation and Schema Setup
  - [x] 1.1 Install and configure PostgreSQL with Docker
  - [x] 1.2 Create database schema following the PRD specifications (market_data, bots, signals, trades, bot_performance tables)
  - [x] 1.3 Implement database connection utility with connection pooling
  - [x] 1.4 Create database models using SQLAlchemy or similar ORM
  - [x] 1.5 Add proper indexes for time-series data optimization
  - [x] 1.6 Setup Redis for pub/sub messaging
  - [x] 1.7 Create database migration scripts and initial data seeding
  - [x] 1.8 Unit test database models and connection utilities
  - [x] 1.9 Add unified logging system we can use for all components

- [ ] 2.0 Market Data Collection and Processing System
  - [x] 2.0.1 Create abstract base class for data collectors with standardized interface, error handling, and data validation
  - [x] 2.0.2 Enhance data collectors with health monitoring, heartbeat system, and auto-restart capabilities
  - [x] 2.0.3 Create collector manager for supervising multiple data collectors with coordinated lifecycle management
  - [x] 2.1 Implement OKX WebSocket API connector for real-time data
  - [x] 2.2 Create OHLCV candle aggregation logic with multiple timeframes (1m, 5m, 15m, 1h, 4h, 1d)
  - [x] 2.3 Build data validation and error handling for market data
  - [x] 2.4 Implement Redis channels for real-time data distribution
  - [x] 2.5 Create data storage layer for OHLCV data in PostgreSQL
  - [x] 2.6 Add technical indicators calculation (SMA, EMA, RSI, MACD, Bollinger Bands)
  - [x] 2.7 Implement data recovery and reconnection logic for API failures (DEFERRED: Basic reconnection exists, comprehensive historical data recovery moved to section 13.0 for future implementation)
  - [x] 2.8 Create data collection service with proper logging
  - [x] 2.9 Unit test data collection and aggregation logic

- [ ] 3.0 Basic Dashboard for Data Visualization and Analysis
  - [x] 3.1 Setup Dash application framework with Mantine UI components
  - [x] 3.2 Create basic layout and navigation structure
  - [x] 3.3 Implement real-time OHLCV price charts with Plotly (candlestick charts)
  - [x] 3.4 Add technical indicators overlay on price charts (SMA, EMA, RSI, MACD)
  - [x] 3.5 Create market data monitoring dashboard (real-time data feed status)
  - [x] 3.6 Build simple data analysis tools (volume analysis, price movement statistics)
  - [x] 3.7 Add the chart time range selector and trigger for realtime data or historical data (when i analyze specified time range i do not want it to reset with realtime data triggers and callbacks)
  - [-] 3.8 Setup real-time dashboard updates using Redis callbacks (DEFERRED: Redis is not used for real-time dashboard updates now)
  - [x] 3.9 Add data export functionality for analysis (CSV/JSON export)
  - [x] 3.10 Unit test basic dashboard components and data visualization

- [ ] 4.0 Strategy Engine Foundation
  - [ ] 4.1 Design and implement `BaseStrategy` abstract class in `strategies/base_strategy.py` with `process_data` and `get_indicators` methods.
  - [ ] 4.2 Implement `EMA Crossover` strategy in `strategies/ema_crossover.py`, inheriting from `BaseStrategy`.
  - [ ] 4.3 Implement `MACD` strategy in `strategies/macd_strategy.py` to provide another reference implementation.
  - [ ] 4.4 Implement `RSI` strategy in `strategies/rsi_strategy.py` for momentum-based signals.
  - [ ] 4.5 Create a strategy factory or registry in `strategies/factory.py` to dynamically load strategies from their configuration files.
  - [ ] 4.6 Implement a JSON-based parameter configuration system in `config/strategies/` for each strategy type.
  - [ ] 4.7 Create comprehensive unit tests in `tests/test_strategies.py` to validate the signal generation logic for each strategy under various market conditions.

- [ ] 5.0 Vectorized Backtesting Engine
  - [ ] 5.1 Design `BacktestingEngine` class in `backtesting/engine.py` to orchestrate the backtesting process.
  - [ ] 5.2 Implement historical data loading from the database using the existing `MarketDataRepository`.
  - [ ] 5.3 Implement the core vectorized backtesting loop using pandas for efficient signal and portfolio calculation.
  - [ ] 5.4 Integrate the strategy factory to run tests on any registered strategy.
  - [ ] 5.5 Create `backtesting/performance.py` to calculate key metrics (Sharpe Ratio, Max Drawdown, Win Rate, Total Return).
  - [ ] 5.6 Implement realistic fee modeling (e.g., 0.1% per trade) and slippage simulation.
  - [ ] 5.7 Define a standardized `BacktestResult` data structure to store trade history, portfolio progression, and final metrics.
  - [ ] 5.8 Create unit tests in `tests/test_backtesting.py` to verify engine calculations and performance metrics against known outcomes.

- [ ] 6.0 Bot Management & Real-Time Simulation Engine
  - [ ] 6.1 Design `BotManager` class in `bot/manager.py` to handle the lifecycle (create, start, stop, monitor) of multiple bot instances.
  - [ ] 6.2 Create a `Bot` class in `bot/instance.py` to encapsulate the state of a single trading bot (config, portfolio, status).
  - [ ] 6.3 Implement a `VirtualPortfolio` class in `bot/portfolio.py` to track virtual assets, balances, and P&L.
  - [ ] 6.4 Develop a simulation loop that processes new market data (initially from the database, mimicking real-time) and triggers strategies.
  - [ ] 6.5 Implement the simulated trade execution logic, updating the `VirtualPortfolio` and recording trades in the database.
  - [ ] 6.6 Implement a heartbeat system where each active bot updates its `last_heartbeat` in the `bots` table.
  - [ ] 6.7 Create a monitoring process within the `BotManager` to check for stalled or crashed bots.
  - [ ] 6.8 Create unit tests in `tests/test_bot_management.py` for bot state transitions, portfolio updates, and trade execution logic.

- [ ] 7.0 Dashboard Integration for Trading Operations
  - [ ] 7.1 Create a new dashboard layout in `dashboard/layouts/trading.py` for bot management and backtesting.
  - [ ] 7.2 Build a bot creation form using Dash Bootstrap Components to select a symbol, strategy, and configuration file.
  - [ ] 7.3 Implement callbacks in `dashboard/callbacks/trading.py` to save new bot configurations to the database.
  - [ ] 7.4 Create a table of all bots from the database, showing their status with "Start/Stop" control buttons.
  - [ ] 7.5 Implement callbacks to trigger the `BotManager` to start and stop bots based on user interaction.
  - [ ] 7.6 Design a simple UI for initiating backtests by selecting a strategy, symbol, and date range.
  - [ ] 7.7 Implement a callback to run the `BacktestingEngine` in a separate process/thread to avoid blocking the UI.

- [ ] 8.0 Portfolio Visualization and Trade Analytics
  - [ ] 8.1 Create a new layout in `dashboard/layouts/performance.py` for displaying backtest and bot performance results.
  - [ ] 8.2 Implement an interactive equity curve chart with Plotly to visualize portfolio value over time.
  - [ ] 8.3 Display key performance metrics (Sharpe Ratio, Drawdown, etc.) in `dbc.Card` components.
  - [ ] 8.4 Add a `dash_ag_grid` or `DataTable` to show detailed trade history.
  - [ ] 8.5 Enhance the main price chart to overlay buy/sell signals from a selected backtest or running bot.
  - [ ] 8.6 Implement callbacks in `dashboard/callbacks/performance.py` to fetch and display results from a completed backtest or an active bot.

- [ ] 9.0 System Finalization and Documentation
  - [ ] 9.1 Write comprehensive documentation in `/docs/guides/` for strategy development and bot configuration.
  - [ ] 9.2 Add detailed docstrings and code comments to all new classes and complex functions.
  - [ ] 9.3 Perform end-to-end integration testing with 5+ bots running concurrently for 24+ hours.
  - [ ] 9.4 Implement comprehensive input validation and error handling on all dashboard components.
  - [ ] 9.5 Create a final deployment checklist and update the main `README.md` with usage instructions.
  - [ ] 9.6 Review and clean up the entire codebase, ensuring consistency with the conventions in `CONTEXT.md`.

- [ ] 10.0 Deployment and Monitoring Setup
  - [ ] 10.1 Create Docker containers for all services
  - [ ] 10.2 Setup docker-compose for local development environment
  - [ ] 10.3 Implement health checks for all services
  - [ ] 10.4 Create deployment scripts and configuration
  - [ ] 10.5 Setup basic logging and monitoring
  - [ ] 10.6 Implement crash recovery and auto-restart mechanisms
  - [ ] 10.7 Create backup and restore procedures for database

- [ ] 11.0 Security and Error Handling
  - [ ] 11.1 Implement secure API key storage and management
  - [ ] 11.2 Add input validation for all user inputs and API responses
  - [ ] 11.3 Create comprehensive error handling and logging throughout system
  - [ ] 11.4 Implement rate limiting for API calls
  - [ ] 11.5 Add data encryption for sensitive information
  - [ ] 11.6 Create security audit checklist and implementation
  - [ ] 11.7 Implement graceful degradation for partial system failures

- [ ] 12.0 Final Integration and Testing
  - [ ] 12.1 Comprehensive system integration testing
  - [ ] 12.2 Performance optimization and bottleneck identification
  - [ ] 12.3 Memory leak detection and cleanup
  - [ ] 12.4 End-to-end testing with multiple concurrent bots
  - [ ] 12.5 Documentation updates and final review
  - [ ] 12.6 Prepare for production deployment
  - [ ] 12.7 Create maintenance and support procedures

- [ ] 13.0 Performance Optimization and Scaling (Future Enhancement)
  - [ ] 13.1 Implement TimescaleDB hypertables for time-series optimization
  - [ ] 13.2 Optimize database schema for hypertable compatibility (composite primary keys)
  - [ ] 13.3 Add database query performance monitoring and analysis
  - [ ] 13.4 Implement advanced connection pooling optimization
  - [ ] 13.5 Add caching layer for frequently accessed market data
  - [ ] 13.6 Optimize data retention and archival strategies
  - [ ] 13.7 Implement horizontal scaling for high-volume trading scenarios
  - [ ] 13.8 Implement comprehensive data recovery with OKX REST API for historical backfill
  - [ ] 13.9 Add gap detection and automatic data recovery during reconnections
  - [ ] 13.10 Implement data integrity validation and conflict resolution for recovered data
  - [ ] 13.11 **Gap-Based Trading Strategy Implementation**
    - [ ] 13.11.1 Implement gap detection algorithms for various timeframes (overnight, weekend, intraday)
    - [ ] 13.11.2 Create gap classification system (breakaway, exhaustion, continuation, common gaps)
    - [ ] 13.11.3 Develop gap-filling probability models using historical data analysis
    - [ ] 13.11.4 Implement gap-based entry/exit strategies with risk management
    - [ ] 13.11.5 Add gap visualization tools in dashboard for manual analysis
    - [ ] 13.11.6 Create automated gap trading bots with configurable parameters
    - [ ] 13.11.7 Implement gap-based backtesting scenarios and performance metrics
    - [ ] 13.11.8 Add gap alert system for real-time gap detection and notification

- [ ] 14.0 Advanced Dashboard Performance and User Experience (Future Enhancement)
  - [ ] 14.1 Implement dashboard state management with browser localStorage persistence
  - [ ] 14.2 Add client-side chart caching to reduce server load and improve responsiveness
  - [ ] 14.3 Implement lazy loading for dashboard components and data-heavy sections
  - [ ] 14.4 Add WebSocket connections for real-time dashboard updates instead of polling
  - [ ] 14.5 Implement dashboard layout customization (draggable panels, custom arrangements)
  - [ ] 14.6 Add multi-threading for callback processing to prevent UI blocking
  - [ ] 14.7 Implement progressive data loading (load recent data first, historical on demand)
  - [ ] 14.8 Add dashboard performance monitoring and bottleneck identification
  - [ ] 14.9 Implement chart virtualization for handling large datasets efficiently
  - [ ] 14.10 Add offline mode capabilities with local data caching
  - [ ] 14.11 Implement smart callback debouncing to reduce unnecessary updates
  - [ ] 14.12 Add dashboard preloading and background data prefetching
  - [ ] 14.13 Implement memory usage optimization for long-running dashboard sessions
  - [ ] 14.14 Add chart export capabilities (PNG, SVG, PDF) with high-quality rendering
  - [ ] 14.15 Implement dashboard mobile responsiveness and touch optimizations
  - [ ] 14.16 **Advanced Gap Analysis Dashboard Features**
    - [ ] 14.16.1 Create dedicated gap analysis panel with gap statistics and trends
    - [ ] 14.16.2 Implement interactive gap charts showing gap size, frequency, and fill rates
    - [ ] 14.16.3 Add gap pattern recognition and historical comparison tools
    - [ ] 14.16.4 Create gap-based alert system with customizable thresholds
    - [ ] 14.16.5 Implement gap trading performance analytics and reporting
    - [ ] 14.16.6 Add gap-based strategy backtesting interface
    - [ ] 14.16.7 Create gap risk assessment tools and position sizing recommendations



### Notes

- **Automatic Database Setup**: Database schema is automatically initialized when Docker containers start via `database/init/` scripts
- **Environment Configuration**: All credentials and settings are managed via `.env` file with consistent defaults
- **Security**: No hardcoded passwords exist in the codebase - all credentials must be loaded from environment variables
- **Clean Schema Approach**: Using `schema_clean.sql` for simpler setup without TimescaleDB hypertables (can be upgraded later)
- Unit tests should be placed in the `tests/` directory with descriptive names
- Use `uv run pytest` to run all tests or `uv run pytest tests/specific_test.py` for individual test files
- JSON configuration files allow rapid strategy parameter testing without code changes
- Redis will be used for real-time messaging between components
- Database models now use JSONB instead of JSON for PostgreSQL optimization
- Connection pooling is configured with proper retry logic and monitoring
- Raw data is stored in PostgreSQL with automatic cleanup utilities (configurable retention period)
- Raw data storage includes: ticker data, trade data, orderbook snapshots, candle data, and balance updates