ec8f5514bb
- Updated all technical indicators to return pandas DataFrames instead of lists, improving consistency and usability. - Modified the `calculate` method in `TechnicalIndicators` to directly return DataFrames with relevant indicator values. - Enhanced the `data_integration.py` to utilize the new DataFrame outputs for better integration with charting. - Updated documentation to reflect the new DataFrame-centric approach, including usage examples and output structures. - Improved error handling to ensure empty DataFrames are returned when insufficient data is available. These changes streamline the indicator calculations and improve the overall architecture, aligning with project standards for maintainability and performance.
236 lines
19 KiB
Markdown
236 lines
19 KiB
Markdown
## Relevant Files
|
|
|
|
- `CONTEXT.md` - **NEW**: Project overview, architecture, tech stack, and implementation status.
|
|
- `app.py` - Main Dash application entry point and dashboard interface
|
|
- `bot_manager.py` - Bot lifecycle management and coordination
|
|
- `database/models.py` - PostgreSQL database models and schema definitions (updated to match schema_clean.sql)
|
|
- `database/schema_clean.sql` - Clean database schema without hypertables (actively used, includes raw_trades table)
|
|
- `database/schema.sql` - Complete database schema with TimescaleDB hypertables (for future optimization)
|
|
- `database/connection.py` - Database connection utility with connection pooling, session management, and raw data utilities
|
|
- `database/redis_manager.py` - Redis connection utility with pub/sub messaging for real-time data distribution
|
|
- `database/migrations/` - Alembic migration system for database schema versioning and updates
|
|
- `database/init/init.sql` - Docker initialization script for automatic database setup
|
|
- `database/init/schema_clean.sql` - Copy of clean schema for Docker initialization
|
|
- `data/base_collector.py` - Abstract base class for all data collectors with standardized interface, error handling, data validation, health monitoring, and auto-restart capabilities
|
|
- `data/collector_manager.py` - Centralized collector management with health monitoring, auto-recovery, and coordinated lifecycle management
|
|
- `data/collection_service.py` - Production-ready data collection service with clean logging, multi-exchange support, and robust error handling
|
|
- `data/__init__.py` - Data collection package initialization
|
|
- `data/exchanges/okx/collector.py` - OKX API integration for real-time market data collection (Corrected Path)
|
|
- `data/aggregator.py` - OHLCV candle aggregation and processing
|
|
- `data/common/indicators.py` - Technical indicators module with SMA, EMA, RSI, MACD, and Bollinger Bands calculations optimized for sparse OHLCV data
|
|
- `strategies/base_strategy.py` - Base strategy class and interface
|
|
- `strategies/ema_crossover.py` - Example EMA crossover strategy implementation
|
|
- `components/dashboard.py` - Dashboard UI components and layouts
|
|
- `components/charts.py` - Price charts and visualization components
|
|
- `backtesting/engine.py` - Backtesting engine for historical strategy testing
|
|
- `backtesting/performance.py` - Performance metrics calculation
|
|
- `config/bot_configs/` - Directory for JSON bot configuration files
|
|
- `config/strategies/` - Directory for JSON strategy parameter files
|
|
- `config/settings.py` - Centralized configuration settings using Pydantic
|
|
- `scripts/dev.py` - Development setup and management script
|
|
- `scripts/start_data_collection.py` - Simple script to start the data collection service with clean output
|
|
- `scripts/production_clean.py` - Clean production OKX data collector script (adapted for service development)
|
|
- `scripts/monitor_clean.py` - Clean database monitor for production data collection status
|
|
- `scripts/init_database.py` - Database initialization and verification script
|
|
- `scripts/test_models.py` - Test script for SQLAlchemy models integration verification
|
|
- `utils/logger.py` - Enhanced unified logging system with verbose console output, automatic cleanup, and configurable retention [USE THIS FOR ALL LOGGING]
|
|
- `alembic.ini` - Alembic configuration for database migrations
|
|
- `requirements.txt` - Python dependencies managed by UV
|
|
- `docker-compose.yml` - Docker services configuration with TimescaleDB support
|
|
- `tests/test_strategies.py` - Unit tests for strategy implementations
|
|
- `tests/test_bot_manager.py` - Unit tests for bot management functionality
|
|
- `tests/test_data_collection.py` - Unit tests for data collection and aggregation
|
|
- `tests/test_data_collection_service.py` - Comprehensive unit tests for the DataCollectionService (25 tests)
|
|
- `tests/test_base_collector.py` - Comprehensive unit tests for the BaseDataCollector abstract class (13 tests)
|
|
- `tests/test_collector_manager.py` - Comprehensive unit tests for the CollectorManager with health monitoring (14 tests)
|
|
- `tests/test_logging_enhanced.py` - Comprehensive unit tests for enhanced logging features (16 tests)
|
|
- `tests/test_indicators.py` - Comprehensive unit tests for technical indicators module (18 tests)
|
|
- `docs/setup.md` - Comprehensive setup guide for new machines and environments
|
|
- `docs/logging.md` - Complete documentation for the enhanced unified logging system
|
|
- `docs/data-collection-service.md` - Complete documentation for the data collection service with usage examples, configuration, and deployment guide
|
|
- `docs/components/technical-indicators.md` - Complete documentation for the technical indicators module with usage examples and integration guide
|
|
- `dashboard/layouts/system_health.py` - Enhanced system health monitoring layout with comprehensive market data monitoring using Mantine components
|
|
- `dashboard/callbacks/system_health.py` - Enhanced system health callbacks with real-time data collection monitoring, database statistics, Redis monitoring, and performance metrics using Mantine components
|
|
|
|
## Tasks
|
|
|
|
- [ ] 1.0 Database Foundation and Schema Setup
|
|
- [x] 1.1 Install and configure PostgreSQL with Docker
|
|
- [x] 1.2 Create database schema following the PRD specifications (market_data, bots, signals, trades, bot_performance tables)
|
|
- [x] 1.3 Implement database connection utility with connection pooling
|
|
- [x] 1.4 Create database models using SQLAlchemy or similar ORM
|
|
- [x] 1.5 Add proper indexes for time-series data optimization
|
|
- [x] 1.6 Setup Redis for pub/sub messaging
|
|
- [x] 1.7 Create database migration scripts and initial data seeding
|
|
- [x] 1.8 Unit test database models and connection utilities
|
|
- [x] 1.9 Add unified logging system we can use for all components
|
|
|
|
- [ ] 2.0 Market Data Collection and Processing System
|
|
- [x] 2.0.1 Create abstract base class for data collectors with standardized interface, error handling, and data validation
|
|
- [x] 2.0.2 Enhance data collectors with health monitoring, heartbeat system, and auto-restart capabilities
|
|
- [x] 2.0.3 Create collector manager for supervising multiple data collectors with coordinated lifecycle management
|
|
- [x] 2.1 Implement OKX WebSocket API connector for real-time data
|
|
- [x] 2.2 Create OHLCV candle aggregation logic with multiple timeframes (1m, 5m, 15m, 1h, 4h, 1d)
|
|
- [x] 2.3 Build data validation and error handling for market data
|
|
- [x] 2.4 Implement Redis channels for real-time data distribution
|
|
- [x] 2.5 Create data storage layer for OHLCV data in PostgreSQL
|
|
- [x] 2.6 Add technical indicators calculation (SMA, EMA, RSI, MACD, Bollinger Bands)
|
|
- [x] 2.7 Implement data recovery and reconnection logic for API failures (DEFERRED: Basic reconnection exists, comprehensive historical data recovery moved to section 13.0 for future implementation)
|
|
- [x] 2.8 Create data collection service with proper logging
|
|
- [x] 2.9 Unit test data collection and aggregation logic
|
|
|
|
- [ ] 3.0 Basic Dashboard for Data Visualization and Analysis
|
|
- [x] 3.1 Setup Dash application framework with Mantine UI components
|
|
- [x] 3.2 Create basic layout and navigation structure
|
|
- [x] 3.3 Implement real-time OHLCV price charts with Plotly (candlestick charts)
|
|
- [x] 3.4 Add technical indicators overlay on price charts (SMA, EMA, RSI, MACD)
|
|
- [x] 3.5 Create market data monitoring dashboard (real-time data feed status)
|
|
- [x] 3.6 Build simple data analysis tools (volume analysis, price movement statistics)
|
|
- [x] 3.7 Add the chart time range selector and trigger for realtime data or historical data (when i analyze specified time range i do not want it to reset with realtime data triggers and callbacks)
|
|
- [-] 3.8 Setup real-time dashboard updates using Redis callbacks (DEFERRED: Redis is not used for real-time dashboard updates now)
|
|
- [x] 3.9 Add data export functionality for analysis (CSV/JSON export)
|
|
- [x] 3.10 Unit test basic dashboard components and data visualization
|
|
|
|
- [ ] 4.0 Strategy Engine Foundation
|
|
- [ ] 4.1 Design and implement `BaseStrategy` abstract class in `strategies/base_strategy.py` with `process_data` and `get_indicators` methods.
|
|
- [ ] 4.2 Implement `EMA Crossover` strategy in `strategies/ema_crossover.py`, inheriting from `BaseStrategy`.
|
|
- [ ] 4.3 Implement `MACD` strategy in `strategies/macd_strategy.py` to provide another reference implementation.
|
|
- [ ] 4.4 Implement `RSI` strategy in `strategies/rsi_strategy.py` for momentum-based signals.
|
|
- [ ] 4.5 Create a strategy factory or registry in `strategies/factory.py` to dynamically load strategies from their configuration files.
|
|
- [ ] 4.6 Implement a JSON-based parameter configuration system in `config/strategies/` for each strategy type.
|
|
- [ ] 4.7 Create comprehensive unit tests in `tests/test_strategies.py` to validate the signal generation logic for each strategy under various market conditions.
|
|
|
|
- [ ] 5.0 Vectorized Backtesting Engine
|
|
- [ ] 5.1 Design `BacktestingEngine` class in `backtesting/engine.py` to orchestrate the backtesting process.
|
|
- [ ] 5.2 Implement historical data loading from the database using the existing `MarketDataRepository`.
|
|
- [ ] 5.3 Implement the core vectorized backtesting loop using pandas for efficient signal and portfolio calculation.
|
|
- [ ] 5.4 Integrate the strategy factory to run tests on any registered strategy.
|
|
- [ ] 5.5 Create `backtesting/performance.py` to calculate key metrics (Sharpe Ratio, Max Drawdown, Win Rate, Total Return).
|
|
- [ ] 5.6 Implement realistic fee modeling (e.g., 0.1% per trade) and slippage simulation.
|
|
- [ ] 5.7 Define a standardized `BacktestResult` data structure to store trade history, portfolio progression, and final metrics.
|
|
- [ ] 5.8 Create unit tests in `tests/test_backtesting.py` to verify engine calculations and performance metrics against known outcomes.
|
|
|
|
- [ ] 6.0 Bot Management & Real-Time Simulation Engine
|
|
- [ ] 6.1 Design `BotManager` class in `bot/manager.py` to handle the lifecycle (create, start, stop, monitor) of multiple bot instances.
|
|
- [ ] 6.2 Create a `Bot` class in `bot/instance.py` to encapsulate the state of a single trading bot (config, portfolio, status).
|
|
- [ ] 6.3 Implement a `VirtualPortfolio` class in `bot/portfolio.py` to track virtual assets, balances, and P&L.
|
|
- [ ] 6.4 Develop a simulation loop that processes new market data (initially from the database, mimicking real-time) and triggers strategies.
|
|
- [ ] 6.5 Implement the simulated trade execution logic, updating the `VirtualPortfolio` and recording trades in the database.
|
|
- [ ] 6.6 Implement a heartbeat system where each active bot updates its `last_heartbeat` in the `bots` table.
|
|
- [ ] 6.7 Create a monitoring process within the `BotManager` to check for stalled or crashed bots.
|
|
- [ ] 6.8 Create unit tests in `tests/test_bot_management.py` for bot state transitions, portfolio updates, and trade execution logic.
|
|
|
|
- [ ] 7.0 Dashboard Integration for Trading Operations
|
|
- [ ] 7.1 Create a new dashboard layout in `dashboard/layouts/trading.py` for bot management and backtesting.
|
|
- [ ] 7.2 Build a bot creation form using Dash Bootstrap Components to select a symbol, strategy, and configuration file.
|
|
- [ ] 7.3 Implement callbacks in `dashboard/callbacks/trading.py` to save new bot configurations to the database.
|
|
- [ ] 7.4 Create a table of all bots from the database, showing their status with "Start/Stop" control buttons.
|
|
- [ ] 7.5 Implement callbacks to trigger the `BotManager` to start and stop bots based on user interaction.
|
|
- [ ] 7.6 Design a simple UI for initiating backtests by selecting a strategy, symbol, and date range.
|
|
- [ ] 7.7 Implement a callback to run the `BacktestingEngine` in a separate process/thread to avoid blocking the UI.
|
|
|
|
- [ ] 8.0 Portfolio Visualization and Trade Analytics
|
|
- [ ] 8.1 Create a new layout in `dashboard/layouts/performance.py` for displaying backtest and bot performance results.
|
|
- [ ] 8.2 Implement an interactive equity curve chart with Plotly to visualize portfolio value over time.
|
|
- [ ] 8.3 Display key performance metrics (Sharpe Ratio, Drawdown, etc.) in `dbc.Card` components.
|
|
- [ ] 8.4 Add a `dash_ag_grid` or `DataTable` to show detailed trade history.
|
|
- [ ] 8.5 Enhance the main price chart to overlay buy/sell signals from a selected backtest or running bot.
|
|
- [ ] 8.6 Implement callbacks in `dashboard/callbacks/performance.py` to fetch and display results from a completed backtest or an active bot.
|
|
|
|
- [ ] 9.0 System Finalization and Documentation
|
|
- [ ] 9.1 Write comprehensive documentation in `/docs/guides/` for strategy development and bot configuration.
|
|
- [ ] 9.2 Add detailed docstrings and code comments to all new classes and complex functions.
|
|
- [ ] 9.3 Perform end-to-end integration testing with 5+ bots running concurrently for 24+ hours.
|
|
- [ ] 9.4 Implement comprehensive input validation and error handling on all dashboard components.
|
|
- [ ] 9.5 Create a final deployment checklist and update the main `README.md` with usage instructions.
|
|
- [ ] 9.6 Review and clean up the entire codebase, ensuring consistency with the conventions in `CONTEXT.md`.
|
|
|
|
- [ ] 10.0 Deployment and Monitoring Setup
|
|
- [ ] 10.1 Create Docker containers for all services
|
|
- [ ] 10.2 Setup docker-compose for local development environment
|
|
- [ ] 10.3 Implement health checks for all services
|
|
- [ ] 10.4 Create deployment scripts and configuration
|
|
- [ ] 10.5 Setup basic logging and monitoring
|
|
- [ ] 10.6 Implement crash recovery and auto-restart mechanisms
|
|
- [ ] 10.7 Create backup and restore procedures for database
|
|
|
|
- [ ] 11.0 Security and Error Handling
|
|
- [ ] 11.1 Implement secure API key storage and management
|
|
- [ ] 11.2 Add input validation for all user inputs and API responses
|
|
- [ ] 11.3 Create comprehensive error handling and logging throughout system
|
|
- [ ] 11.4 Implement rate limiting for API calls
|
|
- [ ] 11.5 Add data encryption for sensitive information
|
|
- [ ] 11.6 Create security audit checklist and implementation
|
|
- [ ] 11.7 Implement graceful degradation for partial system failures
|
|
|
|
- [ ] 12.0 Final Integration and Testing
|
|
- [ ] 12.1 Comprehensive system integration testing
|
|
- [ ] 12.2 Performance optimization and bottleneck identification
|
|
- [ ] 12.3 Memory leak detection and cleanup
|
|
- [ ] 12.4 End-to-end testing with multiple concurrent bots
|
|
- [ ] 12.5 Documentation updates and final review
|
|
- [ ] 12.6 Prepare for production deployment
|
|
- [ ] 12.7 Create maintenance and support procedures
|
|
|
|
- [ ] 13.0 Performance Optimization and Scaling (Future Enhancement)
|
|
- [ ] 13.1 Implement TimescaleDB hypertables for time-series optimization
|
|
- [ ] 13.2 Optimize database schema for hypertable compatibility (composite primary keys)
|
|
- [ ] 13.3 Add database query performance monitoring and analysis
|
|
- [ ] 13.4 Implement advanced connection pooling optimization
|
|
- [ ] 13.5 Add caching layer for frequently accessed market data
|
|
- [ ] 13.6 Optimize data retention and archival strategies
|
|
- [ ] 13.7 Implement horizontal scaling for high-volume trading scenarios
|
|
- [ ] 13.8 Implement comprehensive data recovery with OKX REST API for historical backfill
|
|
- [ ] 13.9 Add gap detection and automatic data recovery during reconnections
|
|
- [ ] 13.10 Implement data integrity validation and conflict resolution for recovered data
|
|
- [ ] 13.11 **Gap-Based Trading Strategy Implementation**
|
|
- [ ] 13.11.1 Implement gap detection algorithms for various timeframes (overnight, weekend, intraday)
|
|
- [ ] 13.11.2 Create gap classification system (breakaway, exhaustion, continuation, common gaps)
|
|
- [ ] 13.11.3 Develop gap-filling probability models using historical data analysis
|
|
- [ ] 13.11.4 Implement gap-based entry/exit strategies with risk management
|
|
- [ ] 13.11.5 Add gap visualization tools in dashboard for manual analysis
|
|
- [ ] 13.11.6 Create automated gap trading bots with configurable parameters
|
|
- [ ] 13.11.7 Implement gap-based backtesting scenarios and performance metrics
|
|
- [ ] 13.11.8 Add gap alert system for real-time gap detection and notification
|
|
|
|
- [ ] 14.0 Advanced Dashboard Performance and User Experience (Future Enhancement)
|
|
- [ ] 14.1 Implement dashboard state management with browser localStorage persistence
|
|
- [ ] 14.2 Add client-side chart caching to reduce server load and improve responsiveness
|
|
- [ ] 14.3 Implement lazy loading for dashboard components and data-heavy sections
|
|
- [ ] 14.4 Add WebSocket connections for real-time dashboard updates instead of polling
|
|
- [ ] 14.5 Implement dashboard layout customization (draggable panels, custom arrangements)
|
|
- [ ] 14.6 Add multi-threading for callback processing to prevent UI blocking
|
|
- [ ] 14.7 Implement progressive data loading (load recent data first, historical on demand)
|
|
- [ ] 14.8 Add dashboard performance monitoring and bottleneck identification
|
|
- [ ] 14.9 Implement chart virtualization for handling large datasets efficiently
|
|
- [ ] 14.10 Add offline mode capabilities with local data caching
|
|
- [ ] 14.11 Implement smart callback debouncing to reduce unnecessary updates
|
|
- [ ] 14.12 Add dashboard preloading and background data prefetching
|
|
- [ ] 14.13 Implement memory usage optimization for long-running dashboard sessions
|
|
- [ ] 14.14 Add chart export capabilities (PNG, SVG, PDF) with high-quality rendering
|
|
- [ ] 14.15 Implement dashboard mobile responsiveness and touch optimizations
|
|
- [ ] 14.16 **Advanced Gap Analysis Dashboard Features**
|
|
- [ ] 14.16.1 Create dedicated gap analysis panel with gap statistics and trends
|
|
- [ ] 14.16.2 Implement interactive gap charts showing gap size, frequency, and fill rates
|
|
- [ ] 14.16.3 Add gap pattern recognition and historical comparison tools
|
|
- [ ] 14.16.4 Create gap-based alert system with customizable thresholds
|
|
- [ ] 14.16.5 Implement gap trading performance analytics and reporting
|
|
- [ ] 14.16.6 Add gap-based strategy backtesting interface
|
|
- [ ] 14.16.7 Create gap risk assessment tools and position sizing recommendations
|
|
|
|
|
|
|
|
### Notes
|
|
|
|
- **Automatic Database Setup**: Database schema is automatically initialized when Docker containers start via `database/init/` scripts
|
|
- **Environment Configuration**: All credentials and settings are managed via `.env` file with consistent defaults
|
|
- **Security**: No hardcoded passwords exist in the codebase - all credentials must be loaded from environment variables
|
|
- **Clean Schema Approach**: Using `schema_clean.sql` for simpler setup without TimescaleDB hypertables (can be upgraded later)
|
|
- Unit tests should be placed in the `tests/` directory with descriptive names
|
|
- Use `uv run pytest` to run all tests or `uv run pytest tests/specific_test.py` for individual test files
|
|
- JSON configuration files allow rapid strategy parameter testing without code changes
|
|
- Redis will be used for real-time messaging between components
|
|
- Database models now use JSONB instead of JSON for PostgreSQL optimization
|
|
- Connection pooling is configured with proper retry logic and monitoring
|
|
- Raw data is stored in PostgreSQL with automatic cleanup utilities (configurable retention period)
|
|
- Raw data storage includes: ticker data, trade data, orderbook snapshots, candle data, and balance updates |