TCPDashboard/tasks/MAIN-task-list.md

Relevant Files

• CONTEXT.md - NEW: Project overview, architecture, tech stack, and implementation status.
• app.py - Main Dash application entry point and dashboard interface
• bot_manager.py - Bot lifecycle management and coordination
• database/models.py - PostgreSQL database models and schema definitions (updated to match schema_clean.sql)
• database/schema_clean.sql - Clean database schema without hypertables (actively used, includes raw_trades table)
• database/schema.sql - Complete database schema with TimescaleDB hypertables (for future optimization)
• database/connection.py - Database connection utility with connection pooling, session management, and raw data utilities
• database/redis_manager.py - Redis connection utility with pub/sub messaging for real-time data distribution
• database/migrations/ - Alembic migration system for database schema versioning and updates
• database/init/init.sql - Docker initialization script for automatic database setup
• database/init/schema_clean.sql - Copy of clean schema for Docker initialization
• data/base_collector.py - Abstract base class for all data collectors with standardized interface, error handling, data validation, health monitoring, and auto-restart capabilities
• data/collector_manager.py - Centralized collector management with health monitoring, auto-recovery, and coordinated lifecycle management
• data/collection_service.py - Production-ready data collection service with clean logging, multi-exchange support, and robust error handling
• data/__init__.py - Data collection package initialization
• data/exchanges/okx/collector.py - OKX API integration for real-time market data collection (Corrected Path)
• data/aggregator.py - OHLCV candle aggregation and processing
• data/common/indicators.py - Technical indicators module with SMA, EMA, RSI, MACD, and Bollinger Bands calculations optimized for sparse OHLCV data
• strategies/base_strategy.py - Base strategy class and interface
• strategies/ema_crossover.py - Example EMA crossover strategy implementation
• components/dashboard.py - Dashboard UI components and layouts
• components/charts.py - Price charts and visualization components
• backtesting/engine.py - Backtesting engine for historical strategy testing
• backtesting/performance.py - Performance metrics calculation
• config/bot_configs/ - Directory for JSON bot configuration files
• config/strategies/ - Directory for JSON strategy parameter files
• config/settings.py - Centralized configuration settings using Pydantic
• scripts/dev.py - Development setup and management script
• scripts/start_data_collection.py - Simple script to start the data collection service with clean output
• scripts/production_clean.py - Clean production OKX data collector script (adapted for service development)
• scripts/monitor_clean.py - Clean database monitor for production data collection status
• scripts/init_database.py - Database initialization and verification script
• scripts/test_models.py - Test script for SQLAlchemy models integration verification
• utils/logger.py - Enhanced unified logging system with verbose console output, automatic cleanup, and configurable retention [USE THIS FOR ALL LOGGING]
• alembic.ini - Alembic configuration for database migrations
• requirements.txt - Python dependencies managed by UV
• docker-compose.yml - Docker services configuration with TimescaleDB support
• tests/test_strategies.py - Unit tests for strategy implementations
• tests/test_bot_manager.py - Unit tests for bot management functionality
• tests/test_data_collection.py - Unit tests for data collection and aggregation
• tests/test_data_collection_service.py - Comprehensive unit tests for the DataCollectionService (25 tests)
• tests/test_base_collector.py - Comprehensive unit tests for the BaseDataCollector abstract class (13 tests)
• tests/test_collector_manager.py - Comprehensive unit tests for the CollectorManager with health monitoring (14 tests)
• tests/test_logging_enhanced.py - Comprehensive unit tests for enhanced logging features (16 tests)
• tests/test_indicators.py - Comprehensive unit tests for technical indicators module (18 tests)
• docs/setup.md - Comprehensive setup guide for new machines and environments
• docs/logging.md - Complete documentation for the enhanced unified logging system
• docs/data-collection-service.md - Complete documentation for the data collection service with usage examples, configuration, and deployment guide
• docs/components/technical-indicators.md - Complete documentation for the technical indicators module with usage examples and integration guide
• dashboard/layouts/system_health.py - Enhanced system health monitoring layout with comprehensive market data monitoring using Mantine components
• dashboard/callbacks/system_health.py - Enhanced system health callbacks with real-time data collection monitoring, database statistics, Redis monitoring, and performance metrics using Mantine components

Tasks

• 1.0 Database Foundation and Schema Setup

  • 1.1 Install and configure PostgreSQL with Docker
  • 1.2 Create database schema following the PRD specifications (market_data, bots, signals, trades, bot_performance tables)
  • 1.3 Implement database connection utility with connection pooling
  • 1.4 Create database models using SQLAlchemy or similar ORM
  • 1.5 Add proper indexes for time-series data optimization
  • 1.6 Setup Redis for pub/sub messaging
  • 1.7 Create database migration scripts and initial data seeding
  • 1.8 Unit test database models and connection utilities
  • 1.9 Add unified logging system we can use for all components

• 2.0 Market Data Collection and Processing System

  • 2.0.1 Create abstract base class for data collectors with standardized interface, error handling, and data validation
  • 2.0.2 Enhance data collectors with health monitoring, heartbeat system, and auto-restart capabilities
  • 2.0.3 Create collector manager for supervising multiple data collectors with coordinated lifecycle management
  • 2.1 Implement OKX WebSocket API connector for real-time data
  • 2.2 Create OHLCV candle aggregation logic with multiple timeframes (1m, 5m, 15m, 1h, 4h, 1d)
  • 2.3 Build data validation and error handling for market data
  • 2.4 Implement Redis channels for real-time data distribution
  • 2.5 Create data storage layer for OHLCV data in PostgreSQL
  • 2.6 Add technical indicators calculation (SMA, EMA, RSI, MACD, Bollinger Bands)
  • 2.7 Implement data recovery and reconnection logic for API failures (DEFERRED: Basic reconnection exists, comprehensive historical data recovery moved to section 13.0 for future implementation)
  • 2.8 Create data collection service with proper logging
  • 2.9 Unit test data collection and aggregation logic

• 3.0 Basic Dashboard for Data Visualization and Analysis

  • 3.1 Setup Dash application framework with Mantine UI components
  • 3.2 Create basic layout and navigation structure
  • 3.3 Implement real-time OHLCV price charts with Plotly (candlestick charts)
  • 3.4 Add technical indicators overlay on price charts (SMA, EMA, RSI, MACD)
  • 3.5 Create market data monitoring dashboard (real-time data feed status)
  • 3.6 Build simple data analysis tools (volume analysis, price movement statistics)
  • 3.7 Add the chart time range selector and trigger for realtime data or historical data (when i analyze specified time range i do not want it to reset with realtime data triggers and callbacks)
  • [-] 3.8 Setup real-time dashboard updates using Redis callbacks (DEFERRED: Redis is not used for real-time dashboard updates now)
  • 3.9 Add data export functionality for analysis (CSV/JSON export)
  • 3.10 Unit test basic dashboard components and data visualization

• 4.0 Strategy Engine Foundation

  • 4.1 Design and implement BaseStrategy abstract class in strategies/base_strategy.py with process_data and get_indicators methods.
  • 4.2 Implement EMA Crossover strategy in strategies/ema_crossover.py, inheriting from BaseStrategy.
  • 4.3 Implement MACD strategy in strategies/macd_strategy.py to provide another reference implementation.
  • 4.4 Implement RSI strategy in strategies/rsi_strategy.py for momentum-based signals.
  • 4.5 Create a strategy factory or registry in strategies/factory.py to dynamically load strategies from their configuration files.
  • 4.6 Implement a JSON-based parameter configuration system in config/strategies/ for each strategy type.
  • 4.7 Create comprehensive unit tests in tests/test_strategies.py to validate the signal generation logic for each strategy under various market conditions.

• 5.0 Vectorized Backtesting Engine

  • 5.1 Design BacktestingEngine class in backtesting/engine.py to orchestrate the backtesting process.
  • 5.2 Implement historical data loading from the database using the existing MarketDataRepository.
  • 5.3 Implement the core vectorized backtesting loop using pandas for efficient signal and portfolio calculation.
  • 5.4 Integrate the strategy factory to run tests on any registered strategy.
  • 5.5 Create backtesting/performance.py to calculate key metrics (Sharpe Ratio, Max Drawdown, Win Rate, Total Return).
  • 5.6 Implement realistic fee modeling (e.g., 0.1% per trade) and slippage simulation.
  • 5.7 Define a standardized BacktestResult data structure to store trade history, portfolio progression, and final metrics.
  • 5.8 Create unit tests in tests/test_backtesting.py to verify engine calculations and performance metrics against known outcomes.

• 6.0 Bot Management & Real-Time Simulation Engine

  • 6.1 Design BotManager class in bot/manager.py to handle the lifecycle (create, start, stop, monitor) of multiple bot instances.
  • 6.2 Create a Bot class in bot/instance.py to encapsulate the state of a single trading bot (config, portfolio, status).
  • 6.3 Implement a VirtualPortfolio class in bot/portfolio.py to track virtual assets, balances, and P&L.
  • 6.4 Develop a simulation loop that processes new market data (initially from the database, mimicking real-time) and triggers strategies.
  • 6.5 Implement the simulated trade execution logic, updating the VirtualPortfolio and recording trades in the database.
  • 6.6 Implement a heartbeat system where each active bot updates its last_heartbeat in the bots table.
  • 6.7 Create a monitoring process within the BotManager to check for stalled or crashed bots.
  • 6.8 Create unit tests in tests/test_bot_management.py for bot state transitions, portfolio updates, and trade execution logic.

• 7.0 Dashboard Integration for Trading Operations

  • 7.1 Create a new dashboard layout in dashboard/layouts/trading.py for bot management and backtesting.
  • 7.2 Build a bot creation form using Dash Bootstrap Components to select a symbol, strategy, and configuration file.
  • 7.3 Implement callbacks in dashboard/callbacks/trading.py to save new bot configurations to the database.
  • 7.4 Create a table of all bots from the database, showing their status with "Start/Stop" control buttons.
  • 7.5 Implement callbacks to trigger the BotManager to start and stop bots based on user interaction.
  • 7.6 Design a simple UI for initiating backtests by selecting a strategy, symbol, and date range.
  • 7.7 Implement a callback to run the BacktestingEngine in a separate process/thread to avoid blocking the UI.

• 8.0 Portfolio Visualization and Trade Analytics

  • 8.1 Create a new layout in dashboard/layouts/performance.py for displaying backtest and bot performance results.
  • 8.2 Implement an interactive equity curve chart with Plotly to visualize portfolio value over time.
  • 8.3 Display key performance metrics (Sharpe Ratio, Drawdown, etc.) in dbc.Card components.
  • 8.4 Add a dash_ag_grid or DataTable to show detailed trade history.
  • 8.5 Enhance the main price chart to overlay buy/sell signals from a selected backtest or running bot.
  • 8.6 Implement callbacks in dashboard/callbacks/performance.py to fetch and display results from a completed backtest or an active bot.

• 9.0 System Finalization and Documentation

  • 9.1 Write comprehensive documentation in /docs/guides/ for strategy development and bot configuration.
  • 9.2 Add detailed docstrings and code comments to all new classes and complex functions.
  • 9.3 Perform end-to-end integration testing with 5+ bots running concurrently for 24+ hours.
  • 9.4 Implement comprehensive input validation and error handling on all dashboard components.
  • 9.5 Create a final deployment checklist and update the main README.md with usage instructions.
  • 9.6 Review and clean up the entire codebase, ensuring consistency with the conventions in CONTEXT.md.

• 10.0 Deployment and Monitoring Setup

  • 10.1 Create Docker containers for all services
  • 10.2 Setup docker-compose for local development environment
  • 10.3 Implement health checks for all services
  • 10.4 Create deployment scripts and configuration
  • 10.5 Setup basic logging and monitoring
  • 10.6 Implement crash recovery and auto-restart mechanisms
  • 10.7 Create backup and restore procedures for database

• 11.0 Security and Error Handling

  • 11.1 Implement secure API key storage and management
  • 11.2 Add input validation for all user inputs and API responses
  • 11.3 Create comprehensive error handling and logging throughout system
  • 11.4 Implement rate limiting for API calls
  • 11.5 Add data encryption for sensitive information
  • 11.6 Create security audit checklist and implementation
  • 11.7 Implement graceful degradation for partial system failures

• 12.0 Final Integration and Testing

  • 12.1 Comprehensive system integration testing
  • 12.2 Performance optimization and bottleneck identification
  • 12.3 Memory leak detection and cleanup
  • 12.4 End-to-end testing with multiple concurrent bots
  • 12.5 Documentation updates and final review
  • 12.6 Prepare for production deployment
  • 12.7 Create maintenance and support procedures

• 13.0 Performance Optimization and Scaling (Future Enhancement)

  • 13.1 Implement TimescaleDB hypertables for time-series optimization
  • 13.2 Optimize database schema for hypertable compatibility (composite primary keys)
  • 13.3 Add database query performance monitoring and analysis
  • 13.4 Implement advanced connection pooling optimization
  • 13.5 Add caching layer for frequently accessed market data
  • 13.6 Optimize data retention and archival strategies
  • 13.7 Implement horizontal scaling for high-volume trading scenarios
  • 13.8 Implement comprehensive data recovery with OKX REST API for historical backfill
  • 13.9 Add gap detection and automatic data recovery during reconnections
  • 13.10 Implement data integrity validation and conflict resolution for recovered data
  • 13.11 Gap-Based Trading Strategy Implementation
    • 13.11.1 Implement gap detection algorithms for various timeframes (overnight, weekend, intraday)
    • 13.11.2 Create gap classification system (breakaway, exhaustion, continuation, common gaps)
    • 13.11.3 Develop gap-filling probability models using historical data analysis
    • 13.11.4 Implement gap-based entry/exit strategies with risk management
    • 13.11.5 Add gap visualization tools in dashboard for manual analysis
    • 13.11.6 Create automated gap trading bots with configurable parameters
    • 13.11.7 Implement gap-based backtesting scenarios and performance metrics
    • 13.11.8 Add gap alert system for real-time gap detection and notification

• 14.0 Advanced Dashboard Performance and User Experience (Future Enhancement)

  • 14.1 Implement dashboard state management with browser localStorage persistence
  • 14.2 Add client-side chart caching to reduce server load and improve responsiveness
  • 14.3 Implement lazy loading for dashboard components and data-heavy sections
  • 14.4 Add WebSocket connections for real-time dashboard updates instead of polling
  • 14.5 Implement dashboard layout customization (draggable panels, custom arrangements)
  • 14.6 Add multi-threading for callback processing to prevent UI blocking
  • 14.7 Implement progressive data loading (load recent data first, historical on demand)
  • 14.8 Add dashboard performance monitoring and bottleneck identification
  • 14.9 Implement chart virtualization for handling large datasets efficiently
  • 14.10 Add offline mode capabilities with local data caching
  • 14.11 Implement smart callback debouncing to reduce unnecessary updates
  • 14.12 Add dashboard preloading and background data prefetching
  • 14.13 Implement memory usage optimization for long-running dashboard sessions
  • 14.14 Add chart export capabilities (PNG, SVG, PDF) with high-quality rendering
  • 14.15 Implement dashboard mobile responsiveness and touch optimizations
  • 14.16 Advanced Gap Analysis Dashboard Features
    • 14.16.1 Create dedicated gap analysis panel with gap statistics and trends
    • 14.16.2 Implement interactive gap charts showing gap size, frequency, and fill rates
    • 14.16.3 Add gap pattern recognition and historical comparison tools
    • 14.16.4 Create gap-based alert system with customizable thresholds
    • 14.16.5 Implement gap trading performance analytics and reporting
    • 14.16.6 Add gap-based strategy backtesting interface
    • 14.16.7 Create gap risk assessment tools and position sizing recommendations

Notes

• Automatic Database Setup: Database schema is automatically initialized when Docker containers start via database/init/ scripts
• Environment Configuration: All credentials and settings are managed via .env file with consistent defaults
• Security: No hardcoded passwords exist in the codebase - all credentials must be loaded from environment variables
• Clean Schema Approach: Using schema_clean.sql for simpler setup without TimescaleDB hypertables (can be upgraded later)
• Unit tests should be placed in the tests/ directory with descriptive names
• Use uv run pytest to run all tests or uv run pytest tests/specific_test.py for individual test files
• JSON configuration files allow rapid strategy parameter testing without code changes
• Redis will be used for real-time messaging between components
• Database models now use JSONB instead of JSON for PostgreSQL optimization
• Connection pooling is configured with proper retry logic and monitoring
• Raw data is stored in PostgreSQL with automatic cleanup utilities (configurable retention period)
• Raw data storage includes: ticker data, trade data, orderbook snapshots, candle data, and balance updates