Implement enhanced data collection system with health monitoring and management
- Introduced `BaseDataCollector` and `CollectorManager` classes for standardized data collection and centralized management. - Added health monitoring features, including auto-restart capabilities and detailed status reporting for collectors. - Updated `env.template` to include new logging and health check configurations. - Enhanced documentation in `docs/data_collectors.md` to provide comprehensive guidance on the new data collection system. - Added unit tests for `BaseDataCollector` and `CollectorManager` to ensure reliability and functionality.
This commit is contained in:
Vasily.onl
228
docs/README.md
Normal file
228
docs/README.md
Normal file
@@ -0,0 +1,228 @@
|
||||
# TCP Dashboard Documentation
|
||||
|
||||
Welcome to the **TCP Dashboard** (Trading Crypto Platform) documentation. This platform provides a comprehensive solution for cryptocurrency trading bot development, backtesting, and portfolio management.
|
||||
|
||||
## 📚 Documentation Index
|
||||
|
||||
### 🏗️ **Architecture & Design**
|
||||
|
||||
- **[Architecture Overview](architecture.md)** - High-level system architecture and component design
|
||||
- **[Project Specification](specification.md)** - Technical specifications and requirements
|
||||
- **[Crypto Bot PRD](crypto-bot-prd.md)** - Product Requirements Document for the crypto trading bot platform
|
||||
|
||||
### 🚀 **Setup & Installation**
|
||||
|
||||
- **[Setup Guide](setup.md)** - Comprehensive setup instructions for new machines and environments
|
||||
- Environment configuration
|
||||
- Database setup with Docker
|
||||
- Development workflow
|
||||
- Production deployment
|
||||
|
||||
### 🔧 **Core Systems**
|
||||
|
||||
#### Data Collection System
|
||||
|
||||
- **[Data Collectors Documentation](data_collectors.md)** - *Comprehensive guide to the enhanced data collector system*
|
||||
- **BaseDataCollector** abstract class with health monitoring
|
||||
- **CollectorManager** for centralized management
|
||||
- Auto-restart and failure recovery
|
||||
- Health monitoring and alerting
|
||||
- Performance optimization
|
||||
- Integration examples
|
||||
- Troubleshooting guide
|
||||
|
||||
#### Logging System
|
||||
|
||||
- **[Enhanced Logging System](logging.md)** - Unified logging framework
|
||||
- Multi-level logging with automatic cleanup
|
||||
- Console and file output with formatting
|
||||
- Performance monitoring
|
||||
- Integration across all components
|
||||
|
||||
## 🎯 **Quick Start**
|
||||
|
||||
1. **New to the platform?** Start with the [Setup Guide](setup.md)
|
||||
2. **Implementing data collectors?** See [Data Collectors Documentation](data_collectors.md)
|
||||
3. **Understanding the architecture?** Read [Architecture Overview](architecture.md)
|
||||
4. **Troubleshooting?** Check component-specific documentation
|
||||
|
||||
## 🏛️ **System Components**
|
||||
|
||||
### Core Infrastructure
|
||||
- **Database Layer**: PostgreSQL with SQLAlchemy models
|
||||
- **Real-time Messaging**: Redis pub/sub for data distribution
|
||||
- **Configuration Management**: Pydantic-based settings
|
||||
- **Containerization**: Docker and docker-compose setup
|
||||
|
||||
### Data Collection & Processing
|
||||
- **Abstract Base Collectors**: Standardized interface for all exchange connectors
|
||||
- **Health Monitoring**: Automatic failure detection and recovery
|
||||
- **Data Validation**: Comprehensive validation for market data
|
||||
- **Multi-Exchange Support**: OKX, Binance, and extensible framework
|
||||
|
||||
### Trading & Strategy Engine
|
||||
- **Strategy Framework**: Base strategy classes and implementations
|
||||
- **Bot Management**: Lifecycle management with JSON configuration
|
||||
- **Backtesting Engine**: Historical strategy testing with performance metrics
|
||||
- **Portfolio Management**: Virtual trading with P&L tracking
|
||||
|
||||
### User Interface
|
||||
- **Dashboard**: Dash-based web interface with Mantine UI
|
||||
- **Real-time Charts**: Interactive price charts with technical indicators
|
||||
- **Bot Controls**: Start/stop/configure trading bots
|
||||
- **Performance Analytics**: Portfolio visualization and trade analytics
|
||||
|
||||
## 📋 **Task Progress**
|
||||
|
||||
The platform follows a structured development approach with clearly defined tasks:
|
||||
|
||||
- ✅ **Database Foundation** - Complete
|
||||
- ✅ **Enhanced Data Collectors** - Complete with health monitoring
|
||||
- ⏳ **Market Data Collection** - In progress (OKX connector next)
|
||||
- ⏳ **Basic Dashboard** - Planned
|
||||
- ⏳ **Strategy Engine** - Planned
|
||||
- ⏳ **Advanced Features** - Planned
|
||||
|
||||
For detailed task tracking, see [tasks/tasks-crypto-bot-prd.md](../tasks/tasks-crypto-bot-prd.md).
|
||||
|
||||
## 🛠️ **Development Workflow**
|
||||
|
||||
### Setting Up Development Environment
|
||||
|
||||
```bash
|
||||
# Clone and setup
|
||||
git clone <repository>
|
||||
cd TCPDashboard
|
||||
|
||||
# Install dependencies with UV
|
||||
uv sync
|
||||
|
||||
# Setup environment
|
||||
cp .env.example .env
|
||||
# Edit .env with your configuration
|
||||
|
||||
# Start services
|
||||
docker-compose up -d
|
||||
|
||||
# Initialize database
|
||||
uv run python scripts/init_database.py
|
||||
|
||||
# Run tests
|
||||
uv run pytest
|
||||
```
|
||||
|
||||
### Key Development Tools
|
||||
|
||||
- **UV**: Modern Python package management
|
||||
- **pytest**: Testing framework with async support
|
||||
- **SQLAlchemy**: Database ORM with migration support
|
||||
- **Dash + Mantine**: Modern web UI framework
|
||||
- **Docker**: Containerized development environment
|
||||
|
||||
## 🔍 **Testing**
|
||||
|
||||
The platform includes comprehensive test coverage:
|
||||
|
||||
- **Unit Tests**: Individual component testing
|
||||
- **Integration Tests**: Cross-component functionality
|
||||
- **Performance Tests**: Load and stress testing
|
||||
- **End-to-End Tests**: Full system workflows
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
uv run pytest
|
||||
|
||||
# Run specific test files
|
||||
uv run pytest tests/test_base_collector.py
|
||||
uv run pytest tests/test_collector_manager.py
|
||||
|
||||
# Run with coverage
|
||||
uv run pytest --cov=data --cov-report=html
|
||||
```
|
||||
|
||||
## 📊 **Monitoring & Observability**
|
||||
|
||||
### Logging
|
||||
- **Structured Logging**: JSON-formatted logs with automatic cleanup
|
||||
- **Multiple Levels**: Debug, Info, Warning, Error with configurable output
|
||||
- **Component Isolation**: Separate loggers for different system components
|
||||
|
||||
### Health Monitoring
|
||||
- **Collector Health**: Real-time status and performance metrics
|
||||
- **Auto-Recovery**: Automatic restart on failures
|
||||
- **Performance Tracking**: Message rates, uptime, error rates
|
||||
|
||||
### Metrics Integration
|
||||
- **Prometheus Support**: Built-in metrics collection
|
||||
- **Custom Dashboards**: System performance visualization
|
||||
- **Alerting**: Configurable alerts for system health
|
||||
|
||||
## 🔐 **Security & Best Practices**
|
||||
|
||||
### Configuration Management
|
||||
- **Environment Variables**: All sensitive data via `.env` files
|
||||
- **No Hardcoded Secrets**: Clean separation of configuration and code
|
||||
- **Validation**: Pydantic-based configuration validation
|
||||
|
||||
### Data Handling
|
||||
- **Input Validation**: Comprehensive validation for all external data
|
||||
- **Error Handling**: Robust error handling with proper logging
|
||||
- **Resource Management**: Proper cleanup and resource management
|
||||
|
||||
### Code Quality
|
||||
- **Type Hints**: Full type annotation coverage
|
||||
- **Documentation**: Comprehensive docstrings and comments
|
||||
- **Testing**: High test coverage with multiple test types
|
||||
- **Code Standards**: Consistent formatting and patterns
|
||||
|
||||
## 🤝 **Contributing**
|
||||
|
||||
### Development Guidelines
|
||||
1. Follow existing code patterns and architecture
|
||||
2. Add comprehensive tests for new functionality
|
||||
3. Update documentation for API changes
|
||||
4. Use type hints and proper error handling
|
||||
5. Follow the existing logging patterns
|
||||
|
||||
### Code Review Process
|
||||
1. Create feature branches from main
|
||||
2. Write tests before implementing features
|
||||
3. Ensure all tests pass and maintain coverage
|
||||
4. Update relevant documentation
|
||||
5. Submit pull requests with clear descriptions
|
||||
|
||||
## 📞 **Support**
|
||||
|
||||
### Getting Help
|
||||
1. **Documentation**: Check relevant component documentation
|
||||
2. **Logs**: Review system logs in `./logs/` directory
|
||||
3. **Status**: Use built-in status and health check methods
|
||||
4. **Tests**: Run test suite to verify system integrity
|
||||
|
||||
### Common Issues
|
||||
- **Database Connection**: Check Docker services and environment variables
|
||||
- **Collector Failures**: Review collector health status and logs
|
||||
- **Performance Issues**: Monitor system resources and optimize accordingly
|
||||
|
||||
---
|
||||
|
||||
## 📁 **File Structure**
|
||||
|
||||
```
|
||||
TCPDashboard/
|
||||
├── docs/ # Documentation (you are here)
|
||||
├── data/ # Data collection system
|
||||
├── database/ # Database models and utilities
|
||||
├── utils/ # Shared utilities (logging, etc.)
|
||||
├── tests/ # Test suite
|
||||
├── examples/ # Usage examples
|
||||
├── config/ # Configuration files
|
||||
├── logs/ # Application logs
|
||||
└── scripts/ # Utility scripts
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
*Last updated: $(date)*
|
||||
|
||||
For the most current information, refer to the individual component documentation linked above.
|
||||
1159
docs/data_collectors.md
Normal file
1159
docs/data_collectors.md
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user