From fa63e7eb2e1f687926fbf72072a493cc130c7bbb Mon Sep 17 00:00:00 2001 From: "Vasily.onl" Date: Sat, 31 May 2025 20:55:52 +0800 Subject: [PATCH] documentation --- docs/README.md | 114 ++++--- docs/architecture/README.md | 41 +++ docs/{ => architecture}/architecture.md | 0 docs/{ => architecture}/crypto-bot-prd.md | 0 docs/components/README.md | 135 ++++++++ docs/{ => components}/data_collectors.md | 0 docs/{ => components}/logging.md | 0 docs/exchanges/README.md | 297 ++++++++++++++++ docs/{ => exchanges}/okx_collector.md | 0 docs/guides/README.md | 309 +++++++++++++++++ docs/{ => guides}/setup.md | 0 docs/reference/README.md | 390 ++++++++++++++++++++++ docs/{ => reference}/specification.md | 0 13 files changed, 1239 insertions(+), 47 deletions(-) create mode 100644 docs/architecture/README.md rename docs/{ => architecture}/architecture.md (100%) rename docs/{ => architecture}/crypto-bot-prd.md (100%) create mode 100644 docs/components/README.md rename docs/{ => components}/data_collectors.md (100%) rename docs/{ => components}/logging.md (100%) create mode 100644 docs/exchanges/README.md rename docs/{ => exchanges}/okx_collector.md (100%) create mode 100644 docs/guides/README.md rename docs/{ => guides}/setup.md (100%) create mode 100644 docs/reference/README.md rename docs/{ => reference}/specification.md (100%) diff --git a/docs/README.md b/docs/README.md index e66bdce..96f1088 100644 --- a/docs/README.md +++ b/docs/README.md @@ -2,59 +2,62 @@ Welcome to the **TCP Dashboard** (Trading Crypto Platform) documentation. This platform provides a comprehensive solution for cryptocurrency trading bot development, backtesting, and portfolio management. -## πŸ“š Documentation Index +## πŸ“š Documentation Structure -### πŸ—οΈ **Architecture & Design** +The documentation is organized into specialized sections for better navigation and maintenance: -- **[Architecture Overview](architecture.md)** - High-level system architecture and component design -- **[Project Specification](specification.md)** - Technical specifications and requirements -- **[Crypto Bot PRD](crypto-bot-prd.md)** - Product Requirements Document for the crypto trading bot platform +### πŸ—οΈ **[Architecture & Design](architecture/)** -### πŸš€ **Setup & Installation** +- **[Architecture Overview](architecture/architecture.md)** - High-level system architecture and component design +- **[Crypto Bot PRD](architecture/crypto-bot-prd.md)** - Product Requirements Document for the crypto trading bot platform -- **[Setup Guide](setup.md)** - Comprehensive setup instructions for new machines and environments - - Environment configuration - - Database setup with Docker - - Development workflow - - Production deployment +### πŸ”§ **[Core Components](components/)** -### πŸ”§ **Core Systems** +- **[Data Collectors](components/data_collectors.md)** - *Comprehensive guide to the enhanced data collector system* + - BaseDataCollector abstract class with health monitoring + - CollectorManager for centralized management + - Exchange Factory Pattern for standardized collector creation + - Modular Exchange Architecture for scalable implementation + - Auto-restart and failure recovery mechanisms -#### Data Collection System +- **[Logging System](components/logging.md)** - *Unified logging framework* + - Multi-level logging with automatic cleanup + - Console and file output with formatting + - Performance monitoring integration -- **[Data Collectors Documentation](data_collectors.md)** - *Comprehensive guide to the enhanced data collector system* - - **BaseDataCollector** abstract class with health monitoring - - **CollectorManager** for centralized management - - **Exchange Factory Pattern** for standardized collector creation - - **Modular Exchange Architecture** for scalable implementation - - Auto-restart and failure recovery - - Health monitoring and alerting - - Performance optimization - - Integration examples - - Troubleshooting guide +### 🌐 **[Exchange Integrations](exchanges/)** -- **[OKX Collector Documentation](okx_collector.md)** - *Complete guide to OKX exchange integration* +- **[OKX Collector](exchanges/okx_collector.md)** - *Complete guide to OKX exchange integration* - Real-time trades, orderbook, and ticker data collection - WebSocket connection management with OKX-specific ping/pong - Factory pattern usage and configuration - - Data processing and validation - - Monitoring and troubleshooting - Production deployment guide -#### Logging System +- **[Exchange Overview](exchanges/)** - Multi-exchange architecture and comparison -- **[Enhanced Logging System](logging.md)** - Unified logging framework - - Multi-level logging with automatic cleanup - - Console and file output with formatting - - Performance monitoring - - Integration across all components +### πŸ“– **[Setup & Guides](guides/)** + +- **[Setup Guide](guides/setup.md)** - *Comprehensive setup instructions* + - Environment configuration and prerequisites + - Database setup with Docker and PostgreSQL + - Development workflow and best practices + - Production deployment guidelines + +### πŸ“‹ **[Technical Reference](reference/)** + +- **[Project Specification](reference/specification.md)** - *Technical specifications and requirements* + - System requirements and constraints + - Database schema specifications + - API endpoint definitions + - Data format specifications ## 🎯 **Quick Start** -1. **New to the platform?** Start with the [Setup Guide](setup.md) -2. **Implementing data collectors?** See [Data Collectors Documentation](data_collectors.md) -3. **Understanding the architecture?** Read [Architecture Overview](architecture.md) -4. **Troubleshooting?** Check component-specific documentation +1. **New to the platform?** Start with the [Setup Guide](guides/setup.md) +2. **Implementing data collectors?** See [Data Collectors Documentation](components/data_collectors.md) +3. **Understanding the architecture?** Read [Architecture Overview](architecture/architecture.md) +4. **Exchange integration?** Check [Exchange Documentation](exchanges/) +5. **Troubleshooting?** Check component-specific documentation ## πŸ›οΈ **System Components** @@ -219,21 +222,38 @@ uv run pytest --cov=data --cov-report=html --- -## πŸ“ **File Structure** +## πŸ“ **Documentation File Structure** ``` -TCPDashboard/ -β”œβ”€β”€ docs/ # Documentation (you are here) -β”œβ”€β”€ data/ # Data collection system -β”œβ”€β”€ database/ # Database models and utilities -β”œβ”€β”€ utils/ # Shared utilities (logging, etc.) -β”œβ”€β”€ tests/ # Test suite -β”œβ”€β”€ examples/ # Usage examples -β”œβ”€β”€ config/ # Configuration files -β”œβ”€β”€ logs/ # Application logs -└── scripts/ # Utility scripts +docs/ +β”œβ”€β”€ README.md # This file - main documentation index +β”œβ”€β”€ architecture/ # System architecture and design +β”‚ β”œβ”€β”€ README.md # Architecture overview +β”‚ β”œβ”€β”€ architecture.md # Technical architecture +β”‚ └── crypto-bot-prd.md # Product requirements +β”œβ”€β”€ components/ # Core system components +β”‚ β”œβ”€β”€ README.md # Component overview +β”‚ β”œβ”€β”€ data_collectors.md # Data collection system +β”‚ └── logging.md # Logging framework +β”œβ”€β”€ exchanges/ # Exchange integrations +β”‚ β”œβ”€β”€ README.md # Exchange overview +β”‚ └── okx_collector.md # OKX implementation +β”œβ”€β”€ guides/ # User guides and tutorials +β”‚ β”œβ”€β”€ README.md # Guide overview +β”‚ └── setup.md # Setup instructions +└── reference/ # Technical reference + β”œβ”€β”€ README.md # Reference overview + └── specification.md # Technical specifications ``` +## πŸ”— **Navigation** + +- **πŸ—οΈ [Architecture & Design](architecture/)** - System design and requirements +- **πŸ”§ [Core Components](components/)** - Technical implementation details +- **🌐 [Exchange Integrations](exchanges/)** - Exchange-specific documentation +- **πŸ“– [Setup & Guides](guides/)** - User guides and tutorials +- **πŸ“‹ [Technical Reference](reference/)** - API specifications and schemas + --- *Last updated: $(date)* diff --git a/docs/architecture/README.md b/docs/architecture/README.md new file mode 100644 index 0000000..8832766 --- /dev/null +++ b/docs/architecture/README.md @@ -0,0 +1,41 @@ +# Architecture Documentation + +This section contains system architecture and design documentation for the TCP Dashboard platform. + +## πŸ“‹ Contents + +### System Architecture + +- **[Architecture Overview](architecture.md)** - *High-level system architecture and component design* + - Core system components and interactions + - Data flow and processing pipelines + - Service architecture and deployment patterns + - Technology stack and infrastructure + +### Product Requirements + +- **[Crypto Bot PRD](crypto-bot-prd.md)** - *Product Requirements Document for the crypto trading bot platform* + - Platform vision and objectives + - Feature specifications and requirements + - User personas and use cases + - Technical requirements and constraints + - Implementation roadmap and milestones + +## πŸ—οΈ System Overview + +The TCP Dashboard follows a modular, microservices-inspired architecture designed for: + +- **Scalability**: Horizontal scaling of individual components +- **Reliability**: Fault tolerance and auto-recovery mechanisms +- **Maintainability**: Clear separation of concerns and modular design +- **Extensibility**: Easy addition of new exchanges, strategies, and features + +## πŸ”— Related Documentation + +- **[Components Documentation](../components/)** - Technical implementation details +- **[Setup Guide](../guides/setup.md)** - System setup and configuration +- **[Reference Documentation](../reference/)** - API specifications and technical references + +--- + +*For the complete documentation index, see the [main documentation README](../README.md).* \ No newline at end of file diff --git a/docs/architecture.md b/docs/architecture/architecture.md similarity index 100% rename from docs/architecture.md rename to docs/architecture/architecture.md diff --git a/docs/crypto-bot-prd.md b/docs/architecture/crypto-bot-prd.md similarity index 100% rename from docs/crypto-bot-prd.md rename to docs/architecture/crypto-bot-prd.md diff --git a/docs/components/README.md b/docs/components/README.md new file mode 100644 index 0000000..6c004fe --- /dev/null +++ b/docs/components/README.md @@ -0,0 +1,135 @@ +# Components Documentation + +This section contains detailed technical documentation for all system components in the TCP Dashboard platform. + +## πŸ“‹ Contents + +### Data Collection System + +- **[Data Collectors](data_collectors.md)** - *Comprehensive guide to the enhanced data collector system* + - **BaseDataCollector** abstract class with health monitoring + - **CollectorManager** for centralized management + - **Exchange Factory Pattern** for standardized collector creation + - **Modular Exchange Architecture** for scalable implementation + - Auto-restart and failure recovery mechanisms + - Health monitoring and alerting systems + - Performance optimization techniques + - Integration examples and patterns + - Comprehensive troubleshooting guide + +### Logging & Monitoring + +- **[Enhanced Logging System](logging.md)** - *Unified logging framework* + - Multi-level logging with automatic cleanup + - Console and file output with formatting + - Performance monitoring integration + - Cross-component logging standards + - Log aggregation and analysis + +## πŸ”§ Component Architecture + +### Core Components + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ CollectorManager β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ Global Health Monitor β”‚ β”‚ +β”‚ β”‚ β€’ System-wide health checks β”‚ β”‚ +β”‚ β”‚ β€’ Auto-restart coordination β”‚ β”‚ +β”‚ β”‚ β€’ Performance analytics β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ OKX Collector β”‚ β”‚Binance Collectorβ”‚ β”‚ Custom β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ Collector β”‚ β”‚ +β”‚ β”‚ β€’ Health Monitorβ”‚ β”‚ β€’ Health Monitorβ”‚ β”‚ β€’ Health Mon β”‚ β”‚ +β”‚ β”‚ β€’ Auto-restart β”‚ β”‚ β€’ Auto-restart β”‚ β”‚ β€’ Auto-resta β”‚ β”‚ +β”‚ β”‚ β€’ Data Validate β”‚ β”‚ β€’ Data Validate β”‚ β”‚ β€’ Data Valid β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### Design Patterns + +- **Factory Pattern**: Standardized component creation across exchanges +- **Observer Pattern**: Event-driven data processing and callbacks +- **Strategy Pattern**: Pluggable data processing strategies +- **Singleton Pattern**: Centralized logging and configuration management + +## πŸš€ Quick Start + +### Using Components + +```python +# Data Collector usage +from data.exchanges import create_okx_collector +from data.base_collector import DataType + +collector = create_okx_collector( + symbol='BTC-USDT', + data_types=[DataType.TRADE, DataType.ORDERBOOK] +) + +# Logging usage +from utils.logger import get_logger + +logger = get_logger("my_component") +logger.info("Component initialized") +``` + +### Component Integration + +```python +# Integrating multiple components +from data.collector_manager import CollectorManager +from utils.logger import get_logger + +manager = CollectorManager("production_system") +logger = get_logger("system_manager") + +# Components work together seamlessly +await manager.start() +logger.info("System started successfully") +``` + +## πŸ“Š Performance & Monitoring + +### Health Monitoring + +All components include built-in health monitoring: + +- **Real-time Status**: Component state and performance metrics +- **Auto-Recovery**: Automatic restart on failures +- **Performance Tracking**: Message rates, uptime, error rates +- **Alerting**: Configurable alerts for component health + +### Logging Integration + +Unified logging across all components: + +- **Structured Logging**: JSON-formatted logs for analysis +- **Multiple Levels**: Debug, Info, Warning, Error levels +- **Automatic Cleanup**: Log rotation and old file cleanup +- **Performance Metrics**: Built-in performance tracking + +## πŸ”— Related Documentation + +- **[Exchange Documentation](../exchanges/)** - Exchange-specific implementations +- **[Architecture Overview](../architecture/)** - System design and patterns +- **[Setup Guide](../guides/setup.md)** - Component configuration and deployment +- **[API Reference](../reference/)** - Technical specifications + +## πŸ“ˆ Future Components + +Planned component additions: + +- **Strategy Engine**: Trading strategy execution framework +- **Portfolio Manager**: Position and risk management +- **Dashboard UI**: Web-based monitoring and control interface +- **Alert Manager**: Advanced alerting and notification system +- **Data Analytics**: Historical data analysis and reporting + +--- + +*For the complete documentation index, see the [main documentation README](../README.md).* \ No newline at end of file diff --git a/docs/data_collectors.md b/docs/components/data_collectors.md similarity index 100% rename from docs/data_collectors.md rename to docs/components/data_collectors.md diff --git a/docs/logging.md b/docs/components/logging.md similarity index 100% rename from docs/logging.md rename to docs/components/logging.md diff --git a/docs/exchanges/README.md b/docs/exchanges/README.md new file mode 100644 index 0000000..c95aada --- /dev/null +++ b/docs/exchanges/README.md @@ -0,0 +1,297 @@ +# Exchange Documentation + +This section contains detailed documentation for all cryptocurrency exchange integrations in the TCP Dashboard platform. + +## πŸ“‹ Contents + +### Supported Exchanges + +#### Production Ready + +- **[OKX Collector](okx_collector.md)** - *Complete guide to OKX exchange integration* + - Real-time trades, orderbook, and ticker data collection + - WebSocket connection management with OKX-specific ping/pong + - Factory pattern usage and configuration + - Data processing and validation + - Monitoring and troubleshooting + - Production deployment guide + +#### Planned Integrations + +- **Binance** - Major global exchange (development planned) +- **Coinbase Pro** - US-regulated exchange (development planned) +- **Kraken** - European exchange (development planned) +- **Bybit** - Derivatives exchange (development planned) + +## πŸ—οΈ Exchange Architecture + +### Modular Design + +Each exchange implementation follows a standardized structure: + +``` +data/exchanges/ +β”œβ”€β”€ __init__.py # Main exports and factory +β”œβ”€β”€ registry.py # Exchange registry and capabilities +β”œβ”€β”€ factory.py # Factory pattern for collectors +└── {exchange}/ # Exchange-specific implementation + β”œβ”€β”€ __init__.py # Exchange exports + β”œβ”€β”€ collector.py # {Exchange}Collector class + └── websocket.py # {Exchange}WebSocketClient class +``` + +### Standardized Interface + +All exchange collectors implement the same interface: + +```python +from data.exchanges import ExchangeFactory, ExchangeCollectorConfig +from data.base_collector import DataType + +# Unified configuration across all exchanges +config = ExchangeCollectorConfig( + exchange='okx', # or 'binance', 'coinbase', etc. + symbol='BTC-USDT', + data_types=[DataType.TRADE, DataType.ORDERBOOK], + auto_restart=True +) + +collector = ExchangeFactory.create_collector(config) +``` + +## πŸš€ Quick Start + +### Using Factory Pattern + +```python +import asyncio +from data.exchanges import get_supported_exchanges, create_okx_collector +from data.base_collector import DataType + +async def main(): + # Check supported exchanges + exchanges = get_supported_exchanges() + print(f"Supported: {exchanges}") # ['okx'] + + # Create OKX collector + collector = create_okx_collector( + symbol='BTC-USDT', + data_types=[DataType.TRADE, DataType.ORDERBOOK] + ) + + # Add data callback + def on_trade(data_point): + print(f"Trade: {data_point.data}") + + collector.add_data_callback(DataType.TRADE, on_trade) + + # Start collection + await collector.start() + await asyncio.sleep(60) + await collector.stop() + +asyncio.run(main()) +``` + +### Multi-Exchange Setup + +```python +from data.exchanges import ExchangeFactory, ExchangeCollectorConfig +from data.collector_manager import CollectorManager + +async def setup_multi_exchange(): + manager = CollectorManager("multi_exchange_system") + + # Future: Multiple exchanges + configs = [ + ExchangeCollectorConfig('okx', 'BTC-USDT', [DataType.TRADE]), + # ExchangeCollectorConfig('binance', 'BTC-USDT', [DataType.TRADE]), + # ExchangeCollectorConfig('coinbase', 'BTC-USD', [DataType.TRADE]) + ] + + for config in configs: + collector = ExchangeFactory.create_collector(config) + manager.add_collector(collector) + + await manager.start() + return manager +``` + +## πŸ“Š Exchange Capabilities + +### Data Types + +Different exchanges support different data types: + +| Exchange | Trades | Orderbook | Ticker | Candles | Balance | +|----------|--------|-----------|--------|---------|---------| +| OKX | βœ… | βœ… | βœ… | πŸ”„ | πŸ”„ | +| Binance | πŸ”„ | πŸ”„ | πŸ”„ | πŸ”„ | πŸ”„ | +| Coinbase | πŸ”„ | πŸ”„ | πŸ”„ | πŸ”„ | πŸ”„ | + +Legend: βœ… Implemented, πŸ”„ Planned, ❌ Not supported + +### Trading Pairs + +Query supported trading pairs for each exchange: + +```python +from data.exchanges import ExchangeFactory + +# Get supported pairs +okx_pairs = ExchangeFactory.get_supported_pairs('okx') +print(f"OKX pairs: {okx_pairs}") + +# Get exchange information +okx_info = ExchangeFactory.get_exchange_info('okx') +print(f"OKX capabilities: {okx_info}") +``` + +## πŸ”§ Exchange Configuration + +### Common Configuration + +All exchanges share common configuration options: + +```python +from data.exchanges import ExchangeCollectorConfig + +config = ExchangeCollectorConfig( + exchange='okx', # Exchange name + symbol='BTC-USDT', # Trading pair + data_types=[DataType.TRADE], # Data types to collect + auto_restart=True, # Auto-restart on failures + health_check_interval=30.0, # Health check interval + store_raw_data=True, # Store raw exchange data + custom_params={ # Exchange-specific parameters + 'ping_interval': 25.0, + 'max_reconnect_attempts': 5 + } +) +``` + +### Exchange-Specific Configuration + +Each exchange has specific configuration files: + +- **OKX**: `config/okx_config.json` +- **Binance**: `config/binance_config.json` (planned) +- **Coinbase**: `config/coinbase_config.json` (planned) + +## πŸ“ˆ Performance Comparison + +### Real-time Data Rates + +Approximate message rates for different exchanges: + +| Exchange | Trades/sec | Orderbook Updates/sec | Latency | +|----------|------------|----------------------|---------| +| OKX | 5-50 | 10-100 | ~50ms | +| Binance | TBD | TBD | TBD | +| Coinbase | TBD | TBD | TBD | + +*Note: Rates vary by trading pair activity* + +### Resource Usage + +Memory and CPU usage per collector: + +| Exchange | Memory (MB) | CPU (%) | Network (KB/s) | +|----------|-------------|---------|----------------| +| OKX | 15-25 | 1-3 | 5-20 | +| Binance | TBD | TBD | TBD | +| Coinbase | TBD | TBD | TBD | + +## πŸ” Monitoring & Debugging + +### Exchange Status + +Monitor exchange-specific metrics: + +```python +# Get exchange status +status = collector.get_status() +print(f"Exchange: {status['exchange']}") +print(f"WebSocket State: {status['websocket_state']}") +print(f"Messages Processed: {status['messages_processed']}") + +# Exchange-specific metrics +if 'websocket_stats' in status: + ws_stats = status['websocket_stats'] + print(f"Reconnections: {ws_stats['reconnections']}") + print(f"Ping/Pong: {ws_stats['pings_sent']}/{ws_stats['pongs_received']}") +``` + +### Debug Mode + +Enable exchange-specific debugging: + +```python +import os +os.environ['LOG_LEVEL'] = 'DEBUG' + +# Detailed exchange logging +collector = create_okx_collector('BTC-USDT', [DataType.TRADE]) +# Check logs: ./logs/okx_collector_btc_usdt_debug.log +``` + +## πŸ› οΈ Adding New Exchanges + +### Implementation Checklist + +To add a new exchange: + +1. **Create Exchange Folder**: `data/exchanges/{exchange}/` +2. **Implement WebSocket Client**: `{exchange}/websocket.py` +3. **Implement Collector**: `{exchange}/collector.py` +4. **Add to Registry**: Update `registry.py` +5. **Create Configuration**: `config/{exchange}_config.json` +6. **Add Documentation**: `docs/exchanges/{exchange}_collector.md` +7. **Add Tests**: `tests/test_{exchange}_collector.py` + +### Implementation Template + +```python +# data/exchanges/newexchange/collector.py +from data.base_collector import BaseDataCollector, DataType +from .websocket import NewExchangeWebSocketClient + +class NewExchangeCollector(BaseDataCollector): + def __init__(self, symbol: str, **kwargs): + super().__init__("newexchange", [symbol], **kwargs) + self.ws_client = NewExchangeWebSocketClient() + + async def connect(self) -> bool: + return await self.ws_client.connect() + + # Implement other required methods... +``` + +## πŸ”— Related Documentation + +- **[Components Documentation](../components/)** - Core system components +- **[Architecture Overview](../architecture/)** - System design +- **[Setup Guide](../guides/setup.md)** - Configuration and deployment +- **[API Reference](../reference/)** - Technical specifications + +## πŸ“ž Support + +### Exchange-Specific Issues + +For exchange-specific problems: + +1. **Check Status**: Use `get_status()` and `get_health_status()` +2. **Review Logs**: Check exchange-specific log files +3. **Verify Configuration**: Confirm exchange configuration files +4. **Test Connection**: Run exchange-specific test scripts + +### Common Issues + +- **Rate Limiting**: Each exchange has different rate limits +- **Symbol Formats**: Trading pair naming conventions vary +- **WebSocket Protocols**: Each exchange has unique WebSocket requirements +- **Data Formats**: Message structures differ between exchanges + +--- + +*For the complete documentation index, see the [main documentation README](../README.md).* \ No newline at end of file diff --git a/docs/okx_collector.md b/docs/exchanges/okx_collector.md similarity index 100% rename from docs/okx_collector.md rename to docs/exchanges/okx_collector.md diff --git a/docs/guides/README.md b/docs/guides/README.md new file mode 100644 index 0000000..3171e0f --- /dev/null +++ b/docs/guides/README.md @@ -0,0 +1,309 @@ +# Guides Documentation + +This section contains user guides, tutorials, and setup instructions for the TCP Dashboard platform. + +## πŸ“‹ Contents + +### Setup & Installation + +- **[Setup Guide](setup.md)** - *Comprehensive setup instructions for new machines and environments* + - Environment configuration and prerequisites + - Database setup with Docker and PostgreSQL + - Development workflow and best practices + - Production deployment guidelines + - Troubleshooting common setup issues + +### Quick Start Guides + +#### For Developers + +```bash +# Quick setup for development +git clone +cd TCPDashboard +uv sync +cp .env.example .env +docker-compose up -d +uv run python scripts/init_database.py +``` + +#### For Users + +```python +# Quick data collection setup +from data.exchanges import create_okx_collector +from data.base_collector import DataType + +collector = create_okx_collector( + symbol='BTC-USDT', + data_types=[DataType.TRADE] +) +await collector.start() +``` + +## πŸš€ Tutorial Series + +### Getting Started + +1. **[Environment Setup](setup.md#environment-setup)** - Setting up your development environment +2. **[First Data Collector](setup.md#first-collector)** - Creating your first data collector +3. **[Database Integration](setup.md#database-setup)** - Connecting to the database +4. **[Adding Monitoring](setup.md#monitoring)** - Setting up logging and monitoring + +### Advanced Topics + +1. **[Multi-Exchange Setup](setup.md#multi-exchange)** - Collecting from multiple exchanges +2. **[Production Deployment](setup.md#production)** - Deploying to production +3. **[Performance Optimization](setup.md#optimization)** - Optimizing for high throughput +4. **[Custom Integrations](setup.md#custom)** - Building custom data sources + +## πŸ› οΈ Development Workflow + +### Daily Development + +```bash +# Start development environment +docker-compose up -d + +# Install new dependencies +uv add package-name + +# Run tests +uv run pytest + +# Check code quality +uv run black . +uv run isort . +``` + +### Code Organization + +- **`data/`**: Data collection and processing +- **`database/`**: Database models and utilities +- **`utils/`**: Shared utilities and logging +- **`tests/`**: Test suite +- **`docs/`**: Documentation +- **`config/`**: Configuration files + +### Best Practices + +1. **Follow existing patterns**: Use established code patterns +2. **Write tests first**: TDD approach for new features +3. **Document changes**: Update docs with code changes +4. **Use type hints**: Full type annotation coverage +5. **Handle errors**: Robust error handling throughout + +## πŸ”§ Configuration Management + +### Environment Variables + +Key environment variables to configure: + +```bash +# Database +DATABASE_URL=postgresql://user:pass@localhost:5432/tcp_dashboard + +# Logging +LOG_LEVEL=INFO +LOG_CLEANUP=true + +# Data Collection +DEFAULT_HEALTH_CHECK_INTERVAL=30 +AUTO_RESTART=true +``` + +### Configuration Files + +The platform uses JSON configuration files: + +- **`config/okx_config.json`**: OKX exchange settings +- **`config/database_config.json`**: Database configuration +- **`config/logging_config.json`**: Logging settings + +### Security Best Practices + +- **Never commit secrets**: Use `.env` files for sensitive data +- **Validate inputs**: Comprehensive input validation +- **Use HTTPS**: Secure connections in production +- **Regular updates**: Keep dependencies updated + +## πŸ“Š Monitoring & Observability + +### Health Monitoring + +The platform includes comprehensive health monitoring: + +```python +# Check system health +from data.collector_manager import CollectorManager + +manager = CollectorManager() +status = manager.get_status() + +print(f"Running collectors: {status['statistics']['running_collectors']}") +print(f"Failed collectors: {status['statistics']['failed_collectors']}") +``` + +### Logging + +Structured logging across all components: + +```python +from utils.logger import get_logger + +logger = get_logger("my_component") +logger.info("Component started", extra={"component": "my_component"}) +``` + +### Performance Metrics + +Built-in performance tracking: + +- **Message rates**: Real-time data processing rates +- **Error rates**: System health and stability +- **Resource usage**: Memory and CPU utilization +- **Uptime**: Component availability metrics + +## πŸ§ͺ Testing + +### Running Tests + +```bash +# Run all tests +uv run pytest + +# Run specific test files +uv run pytest tests/test_base_collector.py + +# Run with coverage +uv run pytest --cov=data --cov-report=html + +# Run integration tests +uv run pytest tests/integration/ +``` + +### Test Organization + +- **Unit tests**: Individual component testing +- **Integration tests**: Cross-component functionality +- **Performance tests**: Load and stress testing +- **End-to-end tests**: Full system workflows + +### Writing Tests + +Follow these patterns when writing tests: + +```python +import pytest +import asyncio +from data.exchanges import create_okx_collector + +@pytest.mark.asyncio +async def test_okx_collector(): + collector = create_okx_collector('BTC-USDT') + assert collector is not None + + # Test lifecycle + await collector.start() + status = collector.get_status() + assert status['status'] == 'running' + + await collector.stop() +``` + +## πŸš€ Deployment + +### Development Deployment + +For local development: + +```bash +# Start services +docker-compose up -d + +# Initialize database +uv run python scripts/init_database.py + +# Start data collection +uv run python scripts/start_collectors.py +``` + +### Production Deployment + +For production environments: + +```bash +# Use production docker-compose +docker-compose -f docker-compose.prod.yml up -d + +# Set production environment +export ENV=production +export LOG_LEVEL=INFO + +# Start with monitoring +uv run python scripts/production_start.py +``` + +### Docker Deployment + +Using Docker containers: + +```dockerfile +FROM python:3.11-slim + +WORKDIR /app +COPY requirements.txt . +RUN pip install -r requirements.txt + +COPY . . +CMD ["python", "-m", "scripts.production_start"] +``` + +## πŸ”— Related Documentation + +- **[Components Documentation](../components/)** - Technical component details +- **[Architecture Overview](../architecture/)** - System design +- **[Exchange Documentation](../exchanges/)** - Exchange integrations +- **[API Reference](../reference/)** - Technical specifications + +## πŸ“ž Support & Troubleshooting + +### Common Issues + +1. **Database Connection Errors** + - Check Docker services: `docker-compose ps` + - Verify environment variables in `.env` + - Test connection: `uv run python scripts/test_db_connection.py` + +2. **Collector Failures** + - Check logs: `tail -f logs/collector_error.log` + - Verify configuration: Review `config/*.json` files + - Test manually: `uv run python scripts/test_okx_collector.py` + +3. **Performance Issues** + - Monitor resource usage: `docker stats` + - Check message rates: Collector status endpoints + - Optimize configuration: Adjust health check intervals + +### Getting Help + +1. **Check Documentation**: Review relevant section documentation +2. **Review Logs**: System logs in `./logs/` directory +3. **Test Components**: Use built-in test scripts +4. **Check Status**: Use status and health check methods + +### Debug Mode + +Enable detailed debugging: + +```bash +export LOG_LEVEL=DEBUG +uv run python your_script.py + +# Check detailed logs +tail -f logs/*_debug.log +``` + +--- + +*For the complete documentation index, see the [main documentation README](../README.md).* \ No newline at end of file diff --git a/docs/setup.md b/docs/guides/setup.md similarity index 100% rename from docs/setup.md rename to docs/guides/setup.md diff --git a/docs/reference/README.md b/docs/reference/README.md new file mode 100644 index 0000000..dd985c9 --- /dev/null +++ b/docs/reference/README.md @@ -0,0 +1,390 @@ +# Reference Documentation + +This section contains technical specifications, API references, and detailed documentation for the TCP Dashboard platform. + +## πŸ“‹ Contents + +### Technical Specifications + +- **[Project Specification](specification.md)** - *Technical specifications and requirements* + - System requirements and constraints + - Database schema specifications + - API endpoint definitions + - Data format specifications + - Integration requirements + +### API References + +#### Data Collection APIs + +```python +# BaseDataCollector API +class BaseDataCollector: + async def start() -> bool + async def stop(force: bool = False) -> None + async def restart() -> bool + def get_status() -> Dict[str, Any] + def get_health_status() -> Dict[str, Any] + def add_data_callback(data_type: DataType, callback: Callable) -> None + +# CollectorManager API +class CollectorManager: + def add_collector(collector: BaseDataCollector) -> None + async def start() -> bool + async def stop() -> None + def get_status() -> Dict[str, Any] + def list_collectors() -> List[str] +``` + +#### Exchange Factory APIs + +```python +# Factory Pattern API +class ExchangeFactory: + @staticmethod + def create_collector(config: ExchangeCollectorConfig) -> BaseDataCollector + + @staticmethod + def create_multiple_collectors(configs: List[ExchangeCollectorConfig]) -> List[BaseDataCollector] + + @staticmethod + def get_supported_exchanges() -> List[str] + + @staticmethod + def validate_config(config: ExchangeCollectorConfig) -> bool + +# Configuration API +@dataclass +class ExchangeCollectorConfig: + exchange: str + symbol: str + data_types: List[DataType] + auto_restart: bool = True + health_check_interval: float = 30.0 + store_raw_data: bool = True + custom_params: Optional[Dict[str, Any]] = None +``` + +## πŸ“Š Data Schemas + +### Market Data Point + +The standardized data structure for all market data: + +```python +@dataclass +class MarketDataPoint: + exchange: str # Exchange name (e.g., 'okx', 'binance') + symbol: str # Trading symbol (e.g., 'BTC-USDT') + timestamp: datetime # Data timestamp (UTC) + data_type: DataType # Type of data (TRADE, ORDERBOOK, etc.) + data: Dict[str, Any] # Raw data payload +``` + +### Data Types + +```python +class DataType(Enum): + TICKER = "ticker" # Price and volume updates + TRADE = "trade" # Individual trade executions + ORDERBOOK = "orderbook" # Order book snapshots + CANDLE = "candle" # OHLCV candle data + BALANCE = "balance" # Account balance updates +``` + +### Status Schemas + +#### Collector Status + +```python +{ + 'exchange': str, # Exchange name + 'status': str, # Current status (running, stopped, error) + 'should_be_running': bool, # Desired state + 'symbols': List[str], # Configured symbols + 'data_types': List[str], # Data types being collected + 'auto_restart': bool, # Auto-restart enabled + 'health': { + 'time_since_heartbeat': float, # Seconds since last heartbeat + 'time_since_data': float, # Seconds since last data + 'max_silence_duration': float # Max allowed silence + }, + 'statistics': { + 'messages_received': int, # Total messages received + 'messages_processed': int, # Successfully processed + 'errors': int, # Error count + 'restarts': int, # Restart count + 'uptime_seconds': float, # Current uptime + 'reconnect_attempts': int, # Current reconnect attempts + 'last_message_time': str, # ISO timestamp + 'connection_uptime': str, # Connection start time + 'last_error': str, # Last error message + 'last_restart_time': str # Last restart time + } +} +``` + +#### Health Status + +```python +{ + 'is_healthy': bool, # Overall health status + 'issues': List[str], # List of current issues + 'status': str, # Current collector status + 'last_heartbeat': str, # Last heartbeat timestamp + 'last_data_received': str, # Last data timestamp + 'should_be_running': bool, # Expected state + 'is_running': bool # Actual running state +} +``` + +## πŸ”§ Configuration Schemas + +### Database Configuration + +```json +{ + "database": { + "url": "postgresql://user:pass@host:port/db", + "pool_size": 10, + "max_overflow": 20, + "pool_timeout": 30, + "pool_recycle": 3600 + }, + "tables": { + "market_data": "market_data", + "raw_trades": "raw_trades", + "collector_status": "collector_status" + } +} +``` + +### Exchange Configuration + +```json +{ + "exchange": "okx", + "connection": { + "public_ws_url": "wss://ws.okx.com:8443/ws/v5/public", + "ping_interval": 25.0, + "pong_timeout": 10.0, + "max_reconnect_attempts": 5, + "reconnect_delay": 5.0 + }, + "data_collection": { + "store_raw_data": true, + "health_check_interval": 30.0, + "auto_restart": true, + "buffer_size": 1000 + }, + "trading_pairs": [ + { + "symbol": "BTC-USDT", + "enabled": true, + "data_types": ["trade", "orderbook"], + "channels": { + "trades": "trades", + "orderbook": "books5", + "ticker": "tickers" + } + } + ] +} +``` + +### Logging Configuration + +```json +{ + "logging": { + "level": "INFO", + "format": "detailed", + "console_output": true, + "file_output": true, + "cleanup": true, + "max_files": 30, + "log_directory": "./logs" + }, + "components": { + "data_collectors": { + "level": "INFO", + "verbose": false + }, + "websocket_clients": { + "level": "DEBUG", + "verbose": true + } + } +} +``` + +## 🌐 Protocol Specifications + +### WebSocket Message Formats + +#### OKX Message Format + +```json +{ + "arg": { + "channel": "trades", + "instId": "BTC-USDT" + }, + "data": [ + { + "instId": "BTC-USDT", + "tradeId": "12345678", + "px": "50000.5", + "sz": "0.001", + "side": "buy", + "ts": "1697123456789" + } + ] +} +``` + +#### Subscription Message Format + +```json +{ + "op": "subscribe", + "args": [ + { + "channel": "trades", + "instId": "BTC-USDT" + }, + { + "channel": "books5", + "instId": "BTC-USDT" + } + ] +} +``` + +### Database Schemas + +#### Market Data Table + +```sql +CREATE TABLE market_data ( + id SERIAL PRIMARY KEY, + exchange VARCHAR(50) NOT NULL, + symbol VARCHAR(50) NOT NULL, + data_type VARCHAR(20) NOT NULL, + timestamp TIMESTAMP WITH TIME ZONE NOT NULL, + data JSONB NOT NULL, + created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + + INDEX(exchange, symbol, timestamp), + INDEX(data_type, timestamp) +); +``` + +#### Raw Trades Table + +```sql +CREATE TABLE raw_trades ( + id SERIAL PRIMARY KEY, + exchange VARCHAR(50) NOT NULL, + symbol VARCHAR(50) NOT NULL, + trade_id VARCHAR(100), + price DECIMAL(20, 8) NOT NULL, + size DECIMAL(20, 8) NOT NULL, + side VARCHAR(10) NOT NULL, + timestamp TIMESTAMP WITH TIME ZONE NOT NULL, + raw_data JSONB, + created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + + UNIQUE(exchange, symbol, trade_id), + INDEX(exchange, symbol, timestamp), + INDEX(timestamp) +); +``` + +## πŸ“ˆ Performance Specifications + +### System Requirements + +#### Minimum Requirements + +- **CPU**: 2 cores, 2.0 GHz +- **Memory**: 4 GB RAM +- **Storage**: 20 GB available space +- **Network**: Stable internet connection (100 Mbps+) + +#### Recommended Requirements + +- **CPU**: 4+ cores, 3.0+ GHz +- **Memory**: 8+ GB RAM +- **Storage**: 100+ GB SSD +- **Network**: High-speed internet (1 Gbps+) + +### Performance Targets + +#### Data Collection + +- **Latency**: < 100ms from exchange to processing +- **Throughput**: 1000+ messages/second per collector +- **Uptime**: 99.9% availability +- **Memory Usage**: < 50 MB per collector + +#### Database Operations + +- **Insert Rate**: 10,000+ inserts/second +- **Query Response**: < 100ms for typical queries +- **Storage Growth**: ~1 GB/month per active trading pair +- **Retention**: 2+ years of historical data + +## πŸ”’ Security Specifications + +### Authentication & Authorization + +- **API Keys**: Secure storage in environment variables +- **Database Access**: Connection pooling with authentication +- **WebSocket Connections**: TLS encryption for all connections +- **Logging**: No sensitive data in logs + +### Data Protection + +- **Encryption**: TLS 1.3 for all external communications +- **Data Validation**: Comprehensive input validation +- **Error Handling**: Secure error messages without data leakage +- **Backup**: Regular automated backups with encryption + +## πŸ”— Related Documentation + +- **[Components Documentation](../components/)** - Implementation details +- **[Architecture Overview](../architecture/)** - System design +- **[Exchange Documentation](../exchanges/)** - Exchange integrations +- **[Setup Guide](../guides/)** - Configuration and deployment + +## πŸ“ž Support + +### API Support + +For API-related questions: + +1. **Check Examples**: Review code examples in each API section +2. **Test Endpoints**: Use provided test scripts +3. **Validate Schemas**: Ensure data matches specified formats +4. **Review Logs**: Check detailed logs for API interactions + +### Schema Validation + +For data schema issues: + +```python +# Validate data point structure +def validate_market_data_point(data_point): + required_fields = ['exchange', 'symbol', 'timestamp', 'data_type', 'data'] + for field in required_fields: + if not hasattr(data_point, field): + raise ValueError(f"Missing required field: {field}") + + if not isinstance(data_point.data_type, DataType): + raise ValueError("Invalid data_type") +``` + +--- + +*For the complete documentation index, see the [main documentation README](../README.md).* \ No newline at end of file diff --git a/docs/specification.md b/docs/reference/specification.md similarity index 100% rename from docs/specification.md rename to docs/reference/specification.md