vasily/TCPDashboard
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Implement enhanced data collection system with health monitoring and management

Browse Source 
- Introduced `BaseDataCollector` and `CollectorManager` classes for standardized data collection and centralized management.
- Added health monitoring features, including auto-restart capabilities and detailed status reporting for collectors.
- Updated `env.template` to include new logging and health check configurations.
- Enhanced documentation in `docs/data_collectors.md` to provide comprehensive guidance on the new data collection system.
- Added unit tests for `BaseDataCollector` and `CollectorManager` to ensure reliability and functionality.
This commit is contained in:
Vasily.onl
2025-05-30 20:33:56 +08:00
parent b7263b023f
commit 4936e5cd73
13 changed files with 4036 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
tasks/tasks-crypto-bot-prd.md
View File

@@ -10,6 +10,9 @@
- `database/migrations/` - Alembic migration system for database schema versioning and updates
- `database/init/init.sql` - Docker initialization script for automatic database setup
- `database/init/schema_clean.sql` - Copy of clean schema for Docker initialization
- `data/base_collector.py` - Abstract base class for all data collectors with standardized interface, error handling, data validation, health monitoring, and auto-restart capabilities
- `data/collector_manager.py` - Centralized collector management with health monitoring, auto-recovery, and coordinated lifecycle management
- `data/__init__.py` - Data collection package initialization
- `data/okx_collector.py` - OKX API integration for real-time market data collection
- `data/aggregator.py` - OHLCV candle aggregation and processing
- `strategies/base_strategy.py` - Base strategy class and interface
@@ -31,6 +34,8 @@
- `tests/test_strategies.py` - Unit tests for strategy implementations
- `tests/test_bot_manager.py` - Unit tests for bot management functionality
- `tests/test_data_collection.py` - Unit tests for data collection and aggregation
- `tests/test_base_collector.py` - Comprehensive unit tests for the BaseDataCollector abstract class (13 tests)
- `tests/test_collector_manager.py` - Comprehensive unit tests for the CollectorManager with health monitoring (14 tests)
- `tests/test_logging_enhanced.py` - Comprehensive unit tests for enhanced logging features (16 tests)
- `docs/setup.md` - Comprehensive setup guide for new machines and environments
- `docs/logging.md` - Complete documentation for the enhanced unified logging system
@@ -49,6 +54,9 @@
- [x] 1.9 Add unified logging system we can use for all components
- [ ] 2.0 Market Data Collection and Processing System
- [x] 2.0.1 Create abstract base class for data collectors with standardized interface, error handling, and data validation
- [x] 2.0.2 Enhance data collectors with health monitoring, heartbeat system, and auto-restart capabilities
- [x] 2.0.3 Create collector manager for supervising multiple data collectors with coordinated lifecycle management
- [ ] 2.1 Implement OKX WebSocket API connector for real-time data
- [ ] 2.2 Create OHLCV candle aggregation logic with multiple timeframes (1m, 5m, 15m, 1h, 4h, 1d)
- [ ] 2.3 Build data validation and error handling for market data