vasily/TCPDashboard
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
TCPDashboard/tasks/MAIN-task-list.md
Vasily.onl 666a58e799 documentation update
2025-06-06 20:33:29 +08:00

18 KiB
Raw Blame History

Relevant Files

  • CONTEXT.md - NEW: Project overview, architecture, tech stack, and implementation status.
  • app.py - Main Dash application entry point and dashboard interface
  • bot_manager.py - Bot lifecycle management and coordination
  • database/models.py - PostgreSQL database models and schema definitions (updated to match schema_clean.sql)
  • database/schema_clean.sql - Clean database schema without hypertables (actively used, includes raw_trades table)
  • database/schema.sql - Complete database schema with TimescaleDB hypertables (for future optimization)
  • database/connection.py - Database connection utility with connection pooling, session management, and raw data utilities
  • database/redis_manager.py - Redis connection utility with pub/sub messaging for real-time data distribution
  • database/migrations/ - Alembic migration system for database schema versioning and updates
  • database/init/init.sql - Docker initialization script for automatic database setup
  • database/init/schema_clean.sql - Copy of clean schema for Docker initialization
  • data/base_collector.py - Abstract base class for all data collectors with standardized interface, error handling, data validation, health monitoring, and auto-restart capabilities
  • data/collector_manager.py - Centralized collector management with health monitoring, auto-recovery, and coordinated lifecycle management
  • data/collection_service.py - Production-ready data collection service with clean logging, multi-exchange support, and robust error handling
  • data/__init__.py - Data collection package initialization
  • data/exchanges/okx/collector.py - OKX API integration for real-time market data collection (Corrected Path)
  • data/aggregator.py - OHLCV candle aggregation and processing
  • data/common/indicators.py - Technical indicators module with SMA, EMA, RSI, MACD, and Bollinger Bands calculations optimized for sparse OHLCV data
  • strategies/base_strategy.py - Base strategy class and interface
  • strategies/ema_crossover.py - Example EMA crossover strategy implementation
  • components/dashboard.py - Dashboard UI components and layouts
  • components/charts.py - Price charts and visualization components
  • backtesting/engine.py - Backtesting engine for historical strategy testing
  • backtesting/performance.py - Performance metrics calculation
  • config/bot_configs/ - Directory for JSON bot configuration files
  • config/strategies/ - Directory for JSON strategy parameter files
  • config/settings.py - Centralized configuration settings using Pydantic
  • scripts/dev.py - Development setup and management script
  • scripts/start_data_collection.py - Simple script to start the data collection service with clean output
  • scripts/production_clean.py - Clean production OKX data collector script (adapted for service development)
  • scripts/monitor_clean.py - Clean database monitor for production data collection status
  • scripts/init_database.py - Database initialization and verification script
  • scripts/test_models.py - Test script for SQLAlchemy models integration verification
  • utils/logger.py - Enhanced unified logging system with verbose console output, automatic cleanup, and configurable retention [USE THIS FOR ALL LOGGING]
  • alembic.ini - Alembic configuration for database migrations
  • requirements.txt - Python dependencies managed by UV
  • docker-compose.yml - Docker services configuration with TimescaleDB support
  • tests/test_strategies.py - Unit tests for strategy implementations
  • tests/test_bot_manager.py - Unit tests for bot management functionality
  • tests/test_data_collection.py - Unit tests for data collection and aggregation
  • tests/test_data_collection_service.py - Comprehensive unit tests for the DataCollectionService (25 tests)
  • tests/test_base_collector.py - Comprehensive unit tests for the BaseDataCollector abstract class (13 tests)
  • tests/test_collector_manager.py - Comprehensive unit tests for the CollectorManager with health monitoring (14 tests)
  • tests/test_logging_enhanced.py - Comprehensive unit tests for enhanced logging features (16 tests)
  • tests/test_indicators.py - Comprehensive unit tests for technical indicators module (18 tests)
  • docs/setup.md - Comprehensive setup guide for new machines and environments
  • docs/logging.md - Complete documentation for the enhanced unified logging system
  • docs/data-collection-service.md - Complete documentation for the data collection service with usage examples, configuration, and deployment guide
  • docs/components/technical-indicators.md - Complete documentation for the technical indicators module with usage examples and integration guide
  • dashboard/layouts/system_health.py - Enhanced system health monitoring layout with comprehensive market data monitoring using Mantine components
  • dashboard/callbacks/system_health.py - Enhanced system health callbacks with real-time data collection monitoring, database statistics, Redis monitoring, and performance metrics using Mantine components

Tasks

  • 1.0 Database Foundation and Schema Setup

    • 1.1 Install and configure PostgreSQL with Docker
    • 1.2 Create database schema following the PRD specifications (market_data, bots, signals, trades, bot_performance tables)
    • 1.3 Implement database connection utility with connection pooling
    • 1.4 Create database models using SQLAlchemy or similar ORM
    • 1.5 Add proper indexes for time-series data optimization
    • 1.6 Setup Redis for pub/sub messaging
    • 1.7 Create database migration scripts and initial data seeding
    • 1.8 Unit test database models and connection utilities
    • 1.9 Add unified logging system we can use for all components

  • 2.0 Market Data Collection and Processing System

    • 2.0.1 Create abstract base class for data collectors with standardized interface, error handling, and data validation
    • 2.0.2 Enhance data collectors with health monitoring, heartbeat system, and auto-restart capabilities
    • 2.0.3 Create collector manager for supervising multiple data collectors with coordinated lifecycle management
    • 2.1 Implement OKX WebSocket API connector for real-time data
    • 2.2 Create OHLCV candle aggregation logic with multiple timeframes (1m, 5m, 15m, 1h, 4h, 1d)
    • 2.3 Build data validation and error handling for market data
    • 2.4 Implement Redis channels for real-time data distribution
    • 2.5 Create data storage layer for OHLCV data in PostgreSQL
    • 2.6 Add technical indicators calculation (SMA, EMA, RSI, MACD, Bollinger Bands)
    • 2.7 Implement data recovery and reconnection logic for API failures (DEFERRED: Basic reconnection exists, comprehensive historical data recovery moved to section 13.0 for future implementation)
    • 2.8 Create data collection service with proper logging
    • 2.9 Unit test data collection and aggregation logic

  • 3.0 Basic Dashboard for Data Visualization and Analysis

    • 3.1 Setup Dash application framework with Mantine UI components
    • 3.2 Create basic layout and navigation structure
    • 3.3 Implement real-time OHLCV price charts with Plotly (candlestick charts)
    • 3.4 Add technical indicators overlay on price charts (SMA, EMA, RSI, MACD)
    • 3.5 Create market data monitoring dashboard (real-time data feed status)
    • 3.6 Build simple data analysis tools (volume analysis, price movement statistics)
    • 3.7 Add the chart time range selector and trigger for realtime data or historical data (when i analyze specified time range i do not want it to reset with realtime data triggers and callbacks)
    • [-] 3.8 Setup real-time dashboard updates using Redis callbacks (DEFERRED: Redis is not used for real-time dashboard updates now)
    • 3.9 Add data export functionality for analysis (CSV/JSON export)
    • 3.10 Unit test basic dashboard components and data visualization

  • 4.0 Strategy Engine Foundation

    • 4.1 Design and implement BaseStrategy abstract class in strategies/base_strategy.py with process_data and get_indicators methods.
    • 4.2 Implement EMA Crossover strategy in strategies/ema_crossover.py, inheriting from BaseStrategy.
    • 4.3 Implement MACD strategy in strategies/macd_strategy.py to provide another reference implementation.
    • 4.4 Implement RSI strategy in strategies/rsi_strategy.py for momentum-based signals.
    • 4.5 Create a strategy factory or registry in strategies/factory.py to dynamically load strategies from their configuration files.
    • 4.6 Implement a JSON-based parameter configuration system in config/strategies/ for each strategy type.
    • 4.7 Create comprehensive unit tests in tests/test_strategies.py to validate the signal generation logic for each strategy under various market conditions.

  • 5.0 Vectorized Backtesting Engine

    • 5.1 Design BacktestingEngine class in backtesting/engine.py to orchestrate the backtesting process.
    • 5.2 Implement historical data loading from the database using the existing MarketDataRepository.
    • 5.3 Implement the core vectorized backtesting loop using pandas for efficient signal and portfolio calculation.
    • 5.4 Integrate the strategy factory to run tests on any registered strategy.
    • 5.5 Create backtesting/performance.py to calculate key metrics (Sharpe Ratio, Max Drawdown, Win Rate, Total Return).
    • 5.6 Implement realistic fee modeling (e.g., 0.1% per trade) and slippage simulation.
    • 5.7 Define a standardized BacktestResult data structure to store trade history, portfolio progression, and final metrics.
    • 5.8 Create unit tests in tests/test_backtesting.py to verify engine calculations and performance metrics against known outcomes.

  • 6.0 Bot Management & Real-Time Simulation Engine

    • 6.1 Design BotManager class in bot/manager.py to handle the lifecycle (create, start, stop, monitor) of multiple bot instances.
    • 6.2 Create a Bot class in bot/instance.py to encapsulate the state of a single trading bot (config, portfolio, status).
    • 6.3 Implement a VirtualPortfolio class in bot/portfolio.py to track virtual assets, balances, and P&L.
    • 6.4 Develop a simulation loop that processes new market data (initially from the database, mimicking real-time) and triggers strategies.
    • 6.5 Implement the simulated trade execution logic, updating the VirtualPortfolio and recording trades in the database.
    • 6.6 Implement a heartbeat system where each active bot updates its last_heartbeat in the bots table.
    • 6.7 Create a monitoring process within the BotManager to check for stalled or crashed bots.
    • 6.8 Create unit tests in tests/test_bot_management.py for bot state transitions, portfolio updates, and trade execution logic.

  • 7.0 Dashboard Integration for Trading Operations

    • 7.1 Create a new dashboard layout in dashboard/layouts/trading.py for bot management and backtesting.
    • 7.2 Build a bot creation form using Dash Bootstrap Components to select a symbol, strategy, and configuration file.
    • 7.3 Implement callbacks in dashboard/callbacks/trading.py to save new bot configurations to the database.
    • 7.4 Create a table of all bots from the database, showing their status with "Start/Stop" control buttons.
    • 7.5 Implement callbacks to trigger the BotManager to start and stop bots based on user interaction.
    • 7.6 Design a simple UI for initiating backtests by selecting a strategy, symbol, and date range.
    • 7.7 Implement a callback to run the BacktestingEngine in a separate process/thread to avoid blocking the UI.

  • 8.0 Portfolio Visualization and Trade Analytics

    • 8.1 Create a new layout in dashboard/layouts/performance.py for displaying backtest and bot performance results.
    • 8.2 Implement an interactive equity curve chart with Plotly to visualize portfolio value over time.
    • 8.3 Display key performance metrics (Sharpe Ratio, Drawdown, etc.) in dbc.Card components.
    • 8.4 Add a dash_ag_grid or DataTable to show detailed trade history.
    • 8.5 Enhance the main price chart to overlay buy/sell signals from a selected backtest or running bot.
    • 8.6 Implement callbacks in dashboard/callbacks/performance.py to fetch and display results from a completed backtest or an active bot.

  • 9.0 System Finalization and Documentation

    • 9.1 Write comprehensive documentation in /docs/guides/ for strategy development and bot configuration.
    • 9.2 Add detailed docstrings and code comments to all new classes and complex functions.
    • 9.3 Perform end-to-end integration testing with 5+ bots running concurrently for 24+ hours.
    • 9.4 Implement comprehensive input validation and error handling on all dashboard components.
    • 9.5 Create a final deployment checklist and update the main README.md with usage instructions.
    • 9.6 Review and clean up the entire codebase, ensuring consistency with the conventions in CONTEXT.md.

  • 10.0 Deployment and Monitoring Setup

    • 10.1 Create Docker containers for all services
    • 10.2 Setup docker-compose for local development environment
    • 10.3 Implement health checks for all services
    • 10.4 Create deployment scripts and configuration
    • 10.5 Setup basic logging and monitoring
    • 10.6 Implement crash recovery and auto-restart mechanisms
    • 10.7 Create backup and restore procedures for database

  • 11.0 Security and Error Handling

    • 11.1 Implement secure API key storage and management
    • 11.2 Add input validation for all user inputs and API responses
    • 11.3 Create comprehensive error handling and logging throughout system
    • 11.4 Implement rate limiting for API calls
    • 11.5 Add data encryption for sensitive information
    • 11.6 Create security audit checklist and implementation
    • 11.7 Implement graceful degradation for partial system failures

  • 12.0 Final Integration and Testing

    • 12.1 Comprehensive system integration testing
    • 12.2 Performance optimization and bottleneck identification
    • 12.3 Memory leak detection and cleanup
    • 12.4 End-to-end testing with multiple concurrent bots
    • 12.5 Documentation updates and final review
    • 12.6 Prepare for production deployment
    • 12.7 Create maintenance and support procedures

  • 13.0 Performance Optimization and Scaling (Future Enhancement)

    • 13.1 Implement TimescaleDB hypertables for time-series optimization
    • 13.2 Optimize database schema for hypertable compatibility (composite primary keys)
    • 13.3 Add database query performance monitoring and analysis
    • 13.4 Implement advanced connection pooling optimization
    • 13.5 Add caching layer for frequently accessed market data
    • 13.6 Optimize data retention and archival strategies
    • 13.7 Implement horizontal scaling for high-volume trading scenarios
    • 13.8 Implement comprehensive data recovery with OKX REST API for historical backfill
    • 13.9 Add gap detection and automatic data recovery during reconnections
    • 13.10 Implement data integrity validation and conflict resolution for recovered data

  • 14.0 Advanced Dashboard Performance and User Experience (Future Enhancement)

    • 14.1 Implement dashboard state management with browser localStorage persistence
    • 14.2 Add client-side chart caching to reduce server load and improve responsiveness
    • 14.3 Implement lazy loading for dashboard components and data-heavy sections
    • 14.4 Add WebSocket connections for real-time dashboard updates instead of polling
    • 14.5 Implement dashboard layout customization (draggable panels, custom arrangements)
    • 14.6 Add multi-threading for callback processing to prevent UI blocking
    • 14.7 Implement progressive data loading (load recent data first, historical on demand)
    • 14.8 Add dashboard performance monitoring and bottleneck identification
    • 14.9 Implement chart virtualization for handling large datasets efficiently
    • 14.10 Add offline mode capabilities with local data caching
    • 14.11 Implement smart callback debouncing to reduce unnecessary updates
    • 14.12 Add dashboard preloading and background data prefetching
    • 14.13 Implement memory usage optimization for long-running dashboard sessions
    • 14.14 Add chart export capabilities (PNG, SVG, PDF) with high-quality rendering
    • 14.15 Implement dashboard mobile responsiveness and touch optimizations

Notes

  • Automatic Database Setup: Database schema is automatically initialized when Docker containers start via database/init/ scripts
  • Environment Configuration: All credentials and settings are managed via .env file with consistent defaults
  • Security: No hardcoded passwords exist in the codebase - all credentials must be loaded from environment variables
  • Clean Schema Approach: Using schema_clean.sql for simpler setup without TimescaleDB hypertables (can be upgraded later)
  • Unit tests should be placed in the tests/ directory with descriptive names
  • Use uv run pytest to run all tests or uv run pytest tests/specific_test.py for individual test files
  • JSON configuration files allow rapid strategy parameter testing without code changes
  • Redis will be used for real-time messaging between components
  • Database models now use JSONB instead of JSON for PostgreSQL optimization
  • Connection pooling is configured with proper retry logic and monitoring
  • Raw data is stored in PostgreSQL with automatic cleanup utilities (configurable retention period)
  • Raw data storage includes: ticker data, trade data, orderbook snapshots, candle data, and balance updates