documentation
This commit is contained in:
Vasily.onl
parent
4510181b39
commit
fa63e7eb2e
114
docs/README.md
114
docs/README.md
@ -2,59 +2,62 @@
|
|||||||
|
|
||||||
Welcome to the **TCP Dashboard** (Trading Crypto Platform) documentation. This platform provides a comprehensive solution for cryptocurrency trading bot development, backtesting, and portfolio management.
|
Welcome to the **TCP Dashboard** (Trading Crypto Platform) documentation. This platform provides a comprehensive solution for cryptocurrency trading bot development, backtesting, and portfolio management.
|
||||||
|
|
||||||
## 📚 Documentation Index
|
## 📚 Documentation Structure
|
||||||
|
|
||||||
### 🏗️ **Architecture & Design**
|
The documentation is organized into specialized sections for better navigation and maintenance:
|
||||||
|
|
||||||
- **[Architecture Overview](architecture.md)** - High-level system architecture and component design
|
### 🏗️ **[Architecture & Design](architecture/)**
|
||||||
- **[Project Specification](specification.md)** - Technical specifications and requirements
|
|
||||||
- **[Crypto Bot PRD](crypto-bot-prd.md)** - Product Requirements Document for the crypto trading bot platform
|
|
||||||
|
|
||||||
### 🚀 **Setup & Installation**
|
- **[Architecture Overview](architecture/architecture.md)** - High-level system architecture and component design
|
||||||
|
- **[Crypto Bot PRD](architecture/crypto-bot-prd.md)** - Product Requirements Document for the crypto trading bot platform
|
||||||
|
|
||||||
- **[Setup Guide](setup.md)** - Comprehensive setup instructions for new machines and environments
|
### 🔧 **[Core Components](components/)**
|
||||||
- Environment configuration
|
|
||||||
- Database setup with Docker
|
|
||||||
- Development workflow
|
|
||||||
- Production deployment
|
|
||||||
|
|
||||||
### 🔧 **Core Systems**
|
- **[Data Collectors](components/data_collectors.md)** - *Comprehensive guide to the enhanced data collector system*
|
||||||
|
- BaseDataCollector abstract class with health monitoring
|
||||||
|
- CollectorManager for centralized management
|
||||||
|
- Exchange Factory Pattern for standardized collector creation
|
||||||
|
- Modular Exchange Architecture for scalable implementation
|
||||||
|
- Auto-restart and failure recovery mechanisms
|
||||||
|
|
||||||
#### Data Collection System
|
- **[Logging System](components/logging.md)** - *Unified logging framework*
|
||||||
|
- Multi-level logging with automatic cleanup
|
||||||
|
- Console and file output with formatting
|
||||||
|
- Performance monitoring integration
|
||||||
|
|
||||||
- **[Data Collectors Documentation](data_collectors.md)** - *Comprehensive guide to the enhanced data collector system*
|
### 🌐 **[Exchange Integrations](exchanges/)**
|
||||||
- **BaseDataCollector** abstract class with health monitoring
|
|
||||||
- **CollectorManager** for centralized management
|
|
||||||
- **Exchange Factory Pattern** for standardized collector creation
|
|
||||||
- **Modular Exchange Architecture** for scalable implementation
|
|
||||||
- Auto-restart and failure recovery
|
|
||||||
- Health monitoring and alerting
|
|
||||||
- Performance optimization
|
|
||||||
- Integration examples
|
|
||||||
- Troubleshooting guide
|
|
||||||
|
|
||||||
- **[OKX Collector Documentation](okx_collector.md)** - *Complete guide to OKX exchange integration*
|
- **[OKX Collector](exchanges/okx_collector.md)** - *Complete guide to OKX exchange integration*
|
||||||
- Real-time trades, orderbook, and ticker data collection
|
- Real-time trades, orderbook, and ticker data collection
|
||||||
- WebSocket connection management with OKX-specific ping/pong
|
- WebSocket connection management with OKX-specific ping/pong
|
||||||
- Factory pattern usage and configuration
|
- Factory pattern usage and configuration
|
||||||
- Data processing and validation
|
|
||||||
- Monitoring and troubleshooting
|
|
||||||
- Production deployment guide
|
- Production deployment guide
|
||||||
|
|
||||||
#### Logging System
|
- **[Exchange Overview](exchanges/)** - Multi-exchange architecture and comparison
|
||||||
|
|
||||||
- **[Enhanced Logging System](logging.md)** - Unified logging framework
|
### 📖 **[Setup & Guides](guides/)**
|
||||||
- Multi-level logging with automatic cleanup
|
|
||||||
- Console and file output with formatting
|
- **[Setup Guide](guides/setup.md)** - *Comprehensive setup instructions*
|
||||||
- Performance monitoring
|
- Environment configuration and prerequisites
|
||||||
- Integration across all components
|
- Database setup with Docker and PostgreSQL
|
||||||
|
- Development workflow and best practices
|
||||||
|
- Production deployment guidelines
|
||||||
|
|
||||||
|
### 📋 **[Technical Reference](reference/)**
|
||||||
|
|
||||||
|
- **[Project Specification](reference/specification.md)** - *Technical specifications and requirements*
|
||||||
|
- System requirements and constraints
|
||||||
|
- Database schema specifications
|
||||||
|
- API endpoint definitions
|
||||||
|
- Data format specifications
|
||||||
|
|
||||||
## 🎯 **Quick Start**
|
## 🎯 **Quick Start**
|
||||||
|
|
||||||
1. **New to the platform?** Start with the [Setup Guide](setup.md)
|
1. **New to the platform?** Start with the [Setup Guide](guides/setup.md)
|
||||||
2. **Implementing data collectors?** See [Data Collectors Documentation](data_collectors.md)
|
2. **Implementing data collectors?** See [Data Collectors Documentation](components/data_collectors.md)
|
||||||
3. **Understanding the architecture?** Read [Architecture Overview](architecture.md)
|
3. **Understanding the architecture?** Read [Architecture Overview](architecture/architecture.md)
|
||||||
4. **Troubleshooting?** Check component-specific documentation
|
4. **Exchange integration?** Check [Exchange Documentation](exchanges/)
|
||||||
|
5. **Troubleshooting?** Check component-specific documentation
|
||||||
|
|
||||||
## 🏛️ **System Components**
|
## 🏛️ **System Components**
|
||||||
|
|
||||||
@ -219,21 +222,38 @@ uv run pytest --cov=data --cov-report=html
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 📁 **File Structure**
|
## 📁 **Documentation File Structure**
|
||||||
|
|
||||||
```
|
```
|
||||||
TCPDashboard/
|
docs/
|
||||||
├── docs/ # Documentation (you are here)
|
├── README.md # This file - main documentation index
|
||||||
├── data/ # Data collection system
|
├── architecture/ # System architecture and design
|
||||||
├── database/ # Database models and utilities
|
│ ├── README.md # Architecture overview
|
||||||
├── utils/ # Shared utilities (logging, etc.)
|
│ ├── architecture.md # Technical architecture
|
||||||
├── tests/ # Test suite
|
│ └── crypto-bot-prd.md # Product requirements
|
||||||
├── examples/ # Usage examples
|
├── components/ # Core system components
|
||||||
├── config/ # Configuration files
|
│ ├── README.md # Component overview
|
||||||
├── logs/ # Application logs
|
│ ├── data_collectors.md # Data collection system
|
||||||
└── scripts/ # Utility scripts
|
│ └── logging.md # Logging framework
|
||||||
|
├── exchanges/ # Exchange integrations
|
||||||
|
│ ├── README.md # Exchange overview
|
||||||
|
│ └── okx_collector.md # OKX implementation
|
||||||
|
├── guides/ # User guides and tutorials
|
||||||
|
│ ├── README.md # Guide overview
|
||||||
|
│ └── setup.md # Setup instructions
|
||||||
|
└── reference/ # Technical reference
|
||||||
|
├── README.md # Reference overview
|
||||||
|
└── specification.md # Technical specifications
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## 🔗 **Navigation**
|
||||||
|
|
||||||
|
- **🏗️ [Architecture & Design](architecture/)** - System design and requirements
|
||||||
|
- **🔧 [Core Components](components/)** - Technical implementation details
|
||||||
|
- **🌐 [Exchange Integrations](exchanges/)** - Exchange-specific documentation
|
||||||
|
- **📖 [Setup & Guides](guides/)** - User guides and tutorials
|
||||||
|
- **📋 [Technical Reference](reference/)** - API specifications and schemas
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
*Last updated: $(date)*
|
*Last updated: $(date)*
|
||||||
|
|||||||
41
docs/architecture/README.md
Normal file
41
docs/architecture/README.md
Normal file
@ -0,0 +1,41 @@
|
|||||||
|
# Architecture Documentation
|
||||||
|
|
||||||
|
This section contains system architecture and design documentation for the TCP Dashboard platform.
|
||||||
|
|
||||||
|
## 📋 Contents
|
||||||
|
|
||||||
|
### System Architecture
|
||||||
|
|
||||||
|
- **[Architecture Overview](architecture.md)** - *High-level system architecture and component design*
|
||||||
|
- Core system components and interactions
|
||||||
|
- Data flow and processing pipelines
|
||||||
|
- Service architecture and deployment patterns
|
||||||
|
- Technology stack and infrastructure
|
||||||
|
|
||||||
|
### Product Requirements
|
||||||
|
|
||||||
|
- **[Crypto Bot PRD](crypto-bot-prd.md)** - *Product Requirements Document for the crypto trading bot platform*
|
||||||
|
- Platform vision and objectives
|
||||||
|
- Feature specifications and requirements
|
||||||
|
- User personas and use cases
|
||||||
|
- Technical requirements and constraints
|
||||||
|
- Implementation roadmap and milestones
|
||||||
|
|
||||||
|
## 🏗️ System Overview
|
||||||
|
|
||||||
|
The TCP Dashboard follows a modular, microservices-inspired architecture designed for:
|
||||||
|
|
||||||
|
- **Scalability**: Horizontal scaling of individual components
|
||||||
|
- **Reliability**: Fault tolerance and auto-recovery mechanisms
|
||||||
|
- **Maintainability**: Clear separation of concerns and modular design
|
||||||
|
- **Extensibility**: Easy addition of new exchanges, strategies, and features
|
||||||
|
|
||||||
|
## 🔗 Related Documentation
|
||||||
|
|
||||||
|
- **[Components Documentation](../components/)** - Technical implementation details
|
||||||
|
- **[Setup Guide](../guides/setup.md)** - System setup and configuration
|
||||||
|
- **[Reference Documentation](../reference/)** - API specifications and technical references
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*For the complete documentation index, see the [main documentation README](../README.md).*
|
||||||
135
docs/components/README.md
Normal file
135
docs/components/README.md
Normal file
@ -0,0 +1,135 @@
|
|||||||
|
# Components Documentation
|
||||||
|
|
||||||
|
This section contains detailed technical documentation for all system components in the TCP Dashboard platform.
|
||||||
|
|
||||||
|
## 📋 Contents
|
||||||
|
|
||||||
|
### Data Collection System
|
||||||
|
|
||||||
|
- **[Data Collectors](data_collectors.md)** - *Comprehensive guide to the enhanced data collector system*
|
||||||
|
- **BaseDataCollector** abstract class with health monitoring
|
||||||
|
- **CollectorManager** for centralized management
|
||||||
|
- **Exchange Factory Pattern** for standardized collector creation
|
||||||
|
- **Modular Exchange Architecture** for scalable implementation
|
||||||
|
- Auto-restart and failure recovery mechanisms
|
||||||
|
- Health monitoring and alerting systems
|
||||||
|
- Performance optimization techniques
|
||||||
|
- Integration examples and patterns
|
||||||
|
- Comprehensive troubleshooting guide
|
||||||
|
|
||||||
|
### Logging & Monitoring
|
||||||
|
|
||||||
|
- **[Enhanced Logging System](logging.md)** - *Unified logging framework*
|
||||||
|
- Multi-level logging with automatic cleanup
|
||||||
|
- Console and file output with formatting
|
||||||
|
- Performance monitoring integration
|
||||||
|
- Cross-component logging standards
|
||||||
|
- Log aggregation and analysis
|
||||||
|
|
||||||
|
## 🔧 Component Architecture
|
||||||
|
|
||||||
|
### Core Components
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────┐
|
||||||
|
│ CollectorManager │
|
||||||
|
│ ┌─────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Global Health Monitor │ │
|
||||||
|
│ │ • System-wide health checks │ │
|
||||||
|
│ │ • Auto-restart coordination │ │
|
||||||
|
│ │ • Performance analytics │ │
|
||||||
|
│ └─────────────────────────────────────────────────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ ┌─────────────────┐ ┌─────────────────┐ ┌──────────────┐ │
|
||||||
|
│ │ OKX Collector │ │Binance Collector│ │ Custom │ │
|
||||||
|
│ │ │ │ │ │ Collector │ │
|
||||||
|
│ │ • Health Monitor│ │ • Health Monitor│ │ • Health Mon │ │
|
||||||
|
│ │ • Auto-restart │ │ • Auto-restart │ │ • Auto-resta │ │
|
||||||
|
│ │ • Data Validate │ │ • Data Validate │ │ • Data Valid │ │
|
||||||
|
│ └─────────────────┘ └─────────────────┘ └──────────────┘ │
|
||||||
|
└─────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Design Patterns
|
||||||
|
|
||||||
|
- **Factory Pattern**: Standardized component creation across exchanges
|
||||||
|
- **Observer Pattern**: Event-driven data processing and callbacks
|
||||||
|
- **Strategy Pattern**: Pluggable data processing strategies
|
||||||
|
- **Singleton Pattern**: Centralized logging and configuration management
|
||||||
|
|
||||||
|
## 🚀 Quick Start
|
||||||
|
|
||||||
|
### Using Components
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Data Collector usage
|
||||||
|
from data.exchanges import create_okx_collector
|
||||||
|
from data.base_collector import DataType
|
||||||
|
|
||||||
|
collector = create_okx_collector(
|
||||||
|
symbol='BTC-USDT',
|
||||||
|
data_types=[DataType.TRADE, DataType.ORDERBOOK]
|
||||||
|
)
|
||||||
|
|
||||||
|
# Logging usage
|
||||||
|
from utils.logger import get_logger
|
||||||
|
|
||||||
|
logger = get_logger("my_component")
|
||||||
|
logger.info("Component initialized")
|
||||||
|
```
|
||||||
|
|
||||||
|
### Component Integration
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Integrating multiple components
|
||||||
|
from data.collector_manager import CollectorManager
|
||||||
|
from utils.logger import get_logger
|
||||||
|
|
||||||
|
manager = CollectorManager("production_system")
|
||||||
|
logger = get_logger("system_manager")
|
||||||
|
|
||||||
|
# Components work together seamlessly
|
||||||
|
await manager.start()
|
||||||
|
logger.info("System started successfully")
|
||||||
|
```
|
||||||
|
|
||||||
|
## 📊 Performance & Monitoring
|
||||||
|
|
||||||
|
### Health Monitoring
|
||||||
|
|
||||||
|
All components include built-in health monitoring:
|
||||||
|
|
||||||
|
- **Real-time Status**: Component state and performance metrics
|
||||||
|
- **Auto-Recovery**: Automatic restart on failures
|
||||||
|
- **Performance Tracking**: Message rates, uptime, error rates
|
||||||
|
- **Alerting**: Configurable alerts for component health
|
||||||
|
|
||||||
|
### Logging Integration
|
||||||
|
|
||||||
|
Unified logging across all components:
|
||||||
|
|
||||||
|
- **Structured Logging**: JSON-formatted logs for analysis
|
||||||
|
- **Multiple Levels**: Debug, Info, Warning, Error levels
|
||||||
|
- **Automatic Cleanup**: Log rotation and old file cleanup
|
||||||
|
- **Performance Metrics**: Built-in performance tracking
|
||||||
|
|
||||||
|
## 🔗 Related Documentation
|
||||||
|
|
||||||
|
- **[Exchange Documentation](../exchanges/)** - Exchange-specific implementations
|
||||||
|
- **[Architecture Overview](../architecture/)** - System design and patterns
|
||||||
|
- **[Setup Guide](../guides/setup.md)** - Component configuration and deployment
|
||||||
|
- **[API Reference](../reference/)** - Technical specifications
|
||||||
|
|
||||||
|
## 📈 Future Components
|
||||||
|
|
||||||
|
Planned component additions:
|
||||||
|
|
||||||
|
- **Strategy Engine**: Trading strategy execution framework
|
||||||
|
- **Portfolio Manager**: Position and risk management
|
||||||
|
- **Dashboard UI**: Web-based monitoring and control interface
|
||||||
|
- **Alert Manager**: Advanced alerting and notification system
|
||||||
|
- **Data Analytics**: Historical data analysis and reporting
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*For the complete documentation index, see the [main documentation README](../README.md).*
|
||||||
297
docs/exchanges/README.md
Normal file
297
docs/exchanges/README.md
Normal file
@ -0,0 +1,297 @@
|
|||||||
|
# Exchange Documentation
|
||||||
|
|
||||||
|
This section contains detailed documentation for all cryptocurrency exchange integrations in the TCP Dashboard platform.
|
||||||
|
|
||||||
|
## 📋 Contents
|
||||||
|
|
||||||
|
### Supported Exchanges
|
||||||
|
|
||||||
|
#### Production Ready
|
||||||
|
|
||||||
|
- **[OKX Collector](okx_collector.md)** - *Complete guide to OKX exchange integration*
|
||||||
|
- Real-time trades, orderbook, and ticker data collection
|
||||||
|
- WebSocket connection management with OKX-specific ping/pong
|
||||||
|
- Factory pattern usage and configuration
|
||||||
|
- Data processing and validation
|
||||||
|
- Monitoring and troubleshooting
|
||||||
|
- Production deployment guide
|
||||||
|
|
||||||
|
#### Planned Integrations
|
||||||
|
|
||||||
|
- **Binance** - Major global exchange (development planned)
|
||||||
|
- **Coinbase Pro** - US-regulated exchange (development planned)
|
||||||
|
- **Kraken** - European exchange (development planned)
|
||||||
|
- **Bybit** - Derivatives exchange (development planned)
|
||||||
|
|
||||||
|
## 🏗️ Exchange Architecture
|
||||||
|
|
||||||
|
### Modular Design
|
||||||
|
|
||||||
|
Each exchange implementation follows a standardized structure:
|
||||||
|
|
||||||
|
```
|
||||||
|
data/exchanges/
|
||||||
|
├── __init__.py # Main exports and factory
|
||||||
|
├── registry.py # Exchange registry and capabilities
|
||||||
|
├── factory.py # Factory pattern for collectors
|
||||||
|
└── {exchange}/ # Exchange-specific implementation
|
||||||
|
├── __init__.py # Exchange exports
|
||||||
|
├── collector.py # {Exchange}Collector class
|
||||||
|
└── websocket.py # {Exchange}WebSocketClient class
|
||||||
|
```
|
||||||
|
|
||||||
|
### Standardized Interface
|
||||||
|
|
||||||
|
All exchange collectors implement the same interface:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from data.exchanges import ExchangeFactory, ExchangeCollectorConfig
|
||||||
|
from data.base_collector import DataType
|
||||||
|
|
||||||
|
# Unified configuration across all exchanges
|
||||||
|
config = ExchangeCollectorConfig(
|
||||||
|
exchange='okx', # or 'binance', 'coinbase', etc.
|
||||||
|
symbol='BTC-USDT',
|
||||||
|
data_types=[DataType.TRADE, DataType.ORDERBOOK],
|
||||||
|
auto_restart=True
|
||||||
|
)
|
||||||
|
|
||||||
|
collector = ExchangeFactory.create_collector(config)
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🚀 Quick Start
|
||||||
|
|
||||||
|
### Using Factory Pattern
|
||||||
|
|
||||||
|
```python
|
||||||
|
import asyncio
|
||||||
|
from data.exchanges import get_supported_exchanges, create_okx_collector
|
||||||
|
from data.base_collector import DataType
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
# Check supported exchanges
|
||||||
|
exchanges = get_supported_exchanges()
|
||||||
|
print(f"Supported: {exchanges}") # ['okx']
|
||||||
|
|
||||||
|
# Create OKX collector
|
||||||
|
collector = create_okx_collector(
|
||||||
|
symbol='BTC-USDT',
|
||||||
|
data_types=[DataType.TRADE, DataType.ORDERBOOK]
|
||||||
|
)
|
||||||
|
|
||||||
|
# Add data callback
|
||||||
|
def on_trade(data_point):
|
||||||
|
print(f"Trade: {data_point.data}")
|
||||||
|
|
||||||
|
collector.add_data_callback(DataType.TRADE, on_trade)
|
||||||
|
|
||||||
|
# Start collection
|
||||||
|
await collector.start()
|
||||||
|
await asyncio.sleep(60)
|
||||||
|
await collector.stop()
|
||||||
|
|
||||||
|
asyncio.run(main())
|
||||||
|
```
|
||||||
|
|
||||||
|
### Multi-Exchange Setup
|
||||||
|
|
||||||
|
```python
|
||||||
|
from data.exchanges import ExchangeFactory, ExchangeCollectorConfig
|
||||||
|
from data.collector_manager import CollectorManager
|
||||||
|
|
||||||
|
async def setup_multi_exchange():
|
||||||
|
manager = CollectorManager("multi_exchange_system")
|
||||||
|
|
||||||
|
# Future: Multiple exchanges
|
||||||
|
configs = [
|
||||||
|
ExchangeCollectorConfig('okx', 'BTC-USDT', [DataType.TRADE]),
|
||||||
|
# ExchangeCollectorConfig('binance', 'BTC-USDT', [DataType.TRADE]),
|
||||||
|
# ExchangeCollectorConfig('coinbase', 'BTC-USD', [DataType.TRADE])
|
||||||
|
]
|
||||||
|
|
||||||
|
for config in configs:
|
||||||
|
collector = ExchangeFactory.create_collector(config)
|
||||||
|
manager.add_collector(collector)
|
||||||
|
|
||||||
|
await manager.start()
|
||||||
|
return manager
|
||||||
|
```
|
||||||
|
|
||||||
|
## 📊 Exchange Capabilities
|
||||||
|
|
||||||
|
### Data Types
|
||||||
|
|
||||||
|
Different exchanges support different data types:
|
||||||
|
|
||||||
|
| Exchange | Trades | Orderbook | Ticker | Candles | Balance |
|
||||||
|
|----------|--------|-----------|--------|---------|---------|
|
||||||
|
| OKX | ✅ | ✅ | ✅ | 🔄 | 🔄 |
|
||||||
|
| Binance | 🔄 | 🔄 | 🔄 | 🔄 | 🔄 |
|
||||||
|
| Coinbase | 🔄 | 🔄 | 🔄 | 🔄 | 🔄 |
|
||||||
|
|
||||||
|
Legend: ✅ Implemented, 🔄 Planned, ❌ Not supported
|
||||||
|
|
||||||
|
### Trading Pairs
|
||||||
|
|
||||||
|
Query supported trading pairs for each exchange:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from data.exchanges import ExchangeFactory
|
||||||
|
|
||||||
|
# Get supported pairs
|
||||||
|
okx_pairs = ExchangeFactory.get_supported_pairs('okx')
|
||||||
|
print(f"OKX pairs: {okx_pairs}")
|
||||||
|
|
||||||
|
# Get exchange information
|
||||||
|
okx_info = ExchangeFactory.get_exchange_info('okx')
|
||||||
|
print(f"OKX capabilities: {okx_info}")
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🔧 Exchange Configuration
|
||||||
|
|
||||||
|
### Common Configuration
|
||||||
|
|
||||||
|
All exchanges share common configuration options:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from data.exchanges import ExchangeCollectorConfig
|
||||||
|
|
||||||
|
config = ExchangeCollectorConfig(
|
||||||
|
exchange='okx', # Exchange name
|
||||||
|
symbol='BTC-USDT', # Trading pair
|
||||||
|
data_types=[DataType.TRADE], # Data types to collect
|
||||||
|
auto_restart=True, # Auto-restart on failures
|
||||||
|
health_check_interval=30.0, # Health check interval
|
||||||
|
store_raw_data=True, # Store raw exchange data
|
||||||
|
custom_params={ # Exchange-specific parameters
|
||||||
|
'ping_interval': 25.0,
|
||||||
|
'max_reconnect_attempts': 5
|
||||||
|
}
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Exchange-Specific Configuration
|
||||||
|
|
||||||
|
Each exchange has specific configuration files:
|
||||||
|
|
||||||
|
- **OKX**: `config/okx_config.json`
|
||||||
|
- **Binance**: `config/binance_config.json` (planned)
|
||||||
|
- **Coinbase**: `config/coinbase_config.json` (planned)
|
||||||
|
|
||||||
|
## 📈 Performance Comparison
|
||||||
|
|
||||||
|
### Real-time Data Rates
|
||||||
|
|
||||||
|
Approximate message rates for different exchanges:
|
||||||
|
|
||||||
|
| Exchange | Trades/sec | Orderbook Updates/sec | Latency |
|
||||||
|
|----------|------------|----------------------|---------|
|
||||||
|
| OKX | 5-50 | 10-100 | ~50ms |
|
||||||
|
| Binance | TBD | TBD | TBD |
|
||||||
|
| Coinbase | TBD | TBD | TBD |
|
||||||
|
|
||||||
|
*Note: Rates vary by trading pair activity*
|
||||||
|
|
||||||
|
### Resource Usage
|
||||||
|
|
||||||
|
Memory and CPU usage per collector:
|
||||||
|
|
||||||
|
| Exchange | Memory (MB) | CPU (%) | Network (KB/s) |
|
||||||
|
|----------|-------------|---------|----------------|
|
||||||
|
| OKX | 15-25 | 1-3 | 5-20 |
|
||||||
|
| Binance | TBD | TBD | TBD |
|
||||||
|
| Coinbase | TBD | TBD | TBD |
|
||||||
|
|
||||||
|
## 🔍 Monitoring & Debugging
|
||||||
|
|
||||||
|
### Exchange Status
|
||||||
|
|
||||||
|
Monitor exchange-specific metrics:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Get exchange status
|
||||||
|
status = collector.get_status()
|
||||||
|
print(f"Exchange: {status['exchange']}")
|
||||||
|
print(f"WebSocket State: {status['websocket_state']}")
|
||||||
|
print(f"Messages Processed: {status['messages_processed']}")
|
||||||
|
|
||||||
|
# Exchange-specific metrics
|
||||||
|
if 'websocket_stats' in status:
|
||||||
|
ws_stats = status['websocket_stats']
|
||||||
|
print(f"Reconnections: {ws_stats['reconnections']}")
|
||||||
|
print(f"Ping/Pong: {ws_stats['pings_sent']}/{ws_stats['pongs_received']}")
|
||||||
|
```
|
||||||
|
|
||||||
|
### Debug Mode
|
||||||
|
|
||||||
|
Enable exchange-specific debugging:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import os
|
||||||
|
os.environ['LOG_LEVEL'] = 'DEBUG'
|
||||||
|
|
||||||
|
# Detailed exchange logging
|
||||||
|
collector = create_okx_collector('BTC-USDT', [DataType.TRADE])
|
||||||
|
# Check logs: ./logs/okx_collector_btc_usdt_debug.log
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🛠️ Adding New Exchanges
|
||||||
|
|
||||||
|
### Implementation Checklist
|
||||||
|
|
||||||
|
To add a new exchange:
|
||||||
|
|
||||||
|
1. **Create Exchange Folder**: `data/exchanges/{exchange}/`
|
||||||
|
2. **Implement WebSocket Client**: `{exchange}/websocket.py`
|
||||||
|
3. **Implement Collector**: `{exchange}/collector.py`
|
||||||
|
4. **Add to Registry**: Update `registry.py`
|
||||||
|
5. **Create Configuration**: `config/{exchange}_config.json`
|
||||||
|
6. **Add Documentation**: `docs/exchanges/{exchange}_collector.md`
|
||||||
|
7. **Add Tests**: `tests/test_{exchange}_collector.py`
|
||||||
|
|
||||||
|
### Implementation Template
|
||||||
|
|
||||||
|
```python
|
||||||
|
# data/exchanges/newexchange/collector.py
|
||||||
|
from data.base_collector import BaseDataCollector, DataType
|
||||||
|
from .websocket import NewExchangeWebSocketClient
|
||||||
|
|
||||||
|
class NewExchangeCollector(BaseDataCollector):
|
||||||
|
def __init__(self, symbol: str, **kwargs):
|
||||||
|
super().__init__("newexchange", [symbol], **kwargs)
|
||||||
|
self.ws_client = NewExchangeWebSocketClient()
|
||||||
|
|
||||||
|
async def connect(self) -> bool:
|
||||||
|
return await self.ws_client.connect()
|
||||||
|
|
||||||
|
# Implement other required methods...
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🔗 Related Documentation
|
||||||
|
|
||||||
|
- **[Components Documentation](../components/)** - Core system components
|
||||||
|
- **[Architecture Overview](../architecture/)** - System design
|
||||||
|
- **[Setup Guide](../guides/setup.md)** - Configuration and deployment
|
||||||
|
- **[API Reference](../reference/)** - Technical specifications
|
||||||
|
|
||||||
|
## 📞 Support
|
||||||
|
|
||||||
|
### Exchange-Specific Issues
|
||||||
|
|
||||||
|
For exchange-specific problems:
|
||||||
|
|
||||||
|
1. **Check Status**: Use `get_status()` and `get_health_status()`
|
||||||
|
2. **Review Logs**: Check exchange-specific log files
|
||||||
|
3. **Verify Configuration**: Confirm exchange configuration files
|
||||||
|
4. **Test Connection**: Run exchange-specific test scripts
|
||||||
|
|
||||||
|
### Common Issues
|
||||||
|
|
||||||
|
- **Rate Limiting**: Each exchange has different rate limits
|
||||||
|
- **Symbol Formats**: Trading pair naming conventions vary
|
||||||
|
- **WebSocket Protocols**: Each exchange has unique WebSocket requirements
|
||||||
|
- **Data Formats**: Message structures differ between exchanges
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*For the complete documentation index, see the [main documentation README](../README.md).*
|
||||||
309
docs/guides/README.md
Normal file
309
docs/guides/README.md
Normal file
@ -0,0 +1,309 @@
|
|||||||
|
# Guides Documentation
|
||||||
|
|
||||||
|
This section contains user guides, tutorials, and setup instructions for the TCP Dashboard platform.
|
||||||
|
|
||||||
|
## 📋 Contents
|
||||||
|
|
||||||
|
### Setup & Installation
|
||||||
|
|
||||||
|
- **[Setup Guide](setup.md)** - *Comprehensive setup instructions for new machines and environments*
|
||||||
|
- Environment configuration and prerequisites
|
||||||
|
- Database setup with Docker and PostgreSQL
|
||||||
|
- Development workflow and best practices
|
||||||
|
- Production deployment guidelines
|
||||||
|
- Troubleshooting common setup issues
|
||||||
|
|
||||||
|
### Quick Start Guides
|
||||||
|
|
||||||
|
#### For Developers
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Quick setup for development
|
||||||
|
git clone <repository>
|
||||||
|
cd TCPDashboard
|
||||||
|
uv sync
|
||||||
|
cp .env.example .env
|
||||||
|
docker-compose up -d
|
||||||
|
uv run python scripts/init_database.py
|
||||||
|
```
|
||||||
|
|
||||||
|
#### For Users
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Quick data collection setup
|
||||||
|
from data.exchanges import create_okx_collector
|
||||||
|
from data.base_collector import DataType
|
||||||
|
|
||||||
|
collector = create_okx_collector(
|
||||||
|
symbol='BTC-USDT',
|
||||||
|
data_types=[DataType.TRADE]
|
||||||
|
)
|
||||||
|
await collector.start()
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🚀 Tutorial Series
|
||||||
|
|
||||||
|
### Getting Started
|
||||||
|
|
||||||
|
1. **[Environment Setup](setup.md#environment-setup)** - Setting up your development environment
|
||||||
|
2. **[First Data Collector](setup.md#first-collector)** - Creating your first data collector
|
||||||
|
3. **[Database Integration](setup.md#database-setup)** - Connecting to the database
|
||||||
|
4. **[Adding Monitoring](setup.md#monitoring)** - Setting up logging and monitoring
|
||||||
|
|
||||||
|
### Advanced Topics
|
||||||
|
|
||||||
|
1. **[Multi-Exchange Setup](setup.md#multi-exchange)** - Collecting from multiple exchanges
|
||||||
|
2. **[Production Deployment](setup.md#production)** - Deploying to production
|
||||||
|
3. **[Performance Optimization](setup.md#optimization)** - Optimizing for high throughput
|
||||||
|
4. **[Custom Integrations](setup.md#custom)** - Building custom data sources
|
||||||
|
|
||||||
|
## 🛠️ Development Workflow
|
||||||
|
|
||||||
|
### Daily Development
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Start development environment
|
||||||
|
docker-compose up -d
|
||||||
|
|
||||||
|
# Install new dependencies
|
||||||
|
uv add package-name
|
||||||
|
|
||||||
|
# Run tests
|
||||||
|
uv run pytest
|
||||||
|
|
||||||
|
# Check code quality
|
||||||
|
uv run black .
|
||||||
|
uv run isort .
|
||||||
|
```
|
||||||
|
|
||||||
|
### Code Organization
|
||||||
|
|
||||||
|
- **`data/`**: Data collection and processing
|
||||||
|
- **`database/`**: Database models and utilities
|
||||||
|
- **`utils/`**: Shared utilities and logging
|
||||||
|
- **`tests/`**: Test suite
|
||||||
|
- **`docs/`**: Documentation
|
||||||
|
- **`config/`**: Configuration files
|
||||||
|
|
||||||
|
### Best Practices
|
||||||
|
|
||||||
|
1. **Follow existing patterns**: Use established code patterns
|
||||||
|
2. **Write tests first**: TDD approach for new features
|
||||||
|
3. **Document changes**: Update docs with code changes
|
||||||
|
4. **Use type hints**: Full type annotation coverage
|
||||||
|
5. **Handle errors**: Robust error handling throughout
|
||||||
|
|
||||||
|
## 🔧 Configuration Management
|
||||||
|
|
||||||
|
### Environment Variables
|
||||||
|
|
||||||
|
Key environment variables to configure:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Database
|
||||||
|
DATABASE_URL=postgresql://user:pass@localhost:5432/tcp_dashboard
|
||||||
|
|
||||||
|
# Logging
|
||||||
|
LOG_LEVEL=INFO
|
||||||
|
LOG_CLEANUP=true
|
||||||
|
|
||||||
|
# Data Collection
|
||||||
|
DEFAULT_HEALTH_CHECK_INTERVAL=30
|
||||||
|
AUTO_RESTART=true
|
||||||
|
```
|
||||||
|
|
||||||
|
### Configuration Files
|
||||||
|
|
||||||
|
The platform uses JSON configuration files:
|
||||||
|
|
||||||
|
- **`config/okx_config.json`**: OKX exchange settings
|
||||||
|
- **`config/database_config.json`**: Database configuration
|
||||||
|
- **`config/logging_config.json`**: Logging settings
|
||||||
|
|
||||||
|
### Security Best Practices
|
||||||
|
|
||||||
|
- **Never commit secrets**: Use `.env` files for sensitive data
|
||||||
|
- **Validate inputs**: Comprehensive input validation
|
||||||
|
- **Use HTTPS**: Secure connections in production
|
||||||
|
- **Regular updates**: Keep dependencies updated
|
||||||
|
|
||||||
|
## 📊 Monitoring & Observability
|
||||||
|
|
||||||
|
### Health Monitoring
|
||||||
|
|
||||||
|
The platform includes comprehensive health monitoring:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Check system health
|
||||||
|
from data.collector_manager import CollectorManager
|
||||||
|
|
||||||
|
manager = CollectorManager()
|
||||||
|
status = manager.get_status()
|
||||||
|
|
||||||
|
print(f"Running collectors: {status['statistics']['running_collectors']}")
|
||||||
|
print(f"Failed collectors: {status['statistics']['failed_collectors']}")
|
||||||
|
```
|
||||||
|
|
||||||
|
### Logging
|
||||||
|
|
||||||
|
Structured logging across all components:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from utils.logger import get_logger
|
||||||
|
|
||||||
|
logger = get_logger("my_component")
|
||||||
|
logger.info("Component started", extra={"component": "my_component"})
|
||||||
|
```
|
||||||
|
|
||||||
|
### Performance Metrics
|
||||||
|
|
||||||
|
Built-in performance tracking:
|
||||||
|
|
||||||
|
- **Message rates**: Real-time data processing rates
|
||||||
|
- **Error rates**: System health and stability
|
||||||
|
- **Resource usage**: Memory and CPU utilization
|
||||||
|
- **Uptime**: Component availability metrics
|
||||||
|
|
||||||
|
## 🧪 Testing
|
||||||
|
|
||||||
|
### Running Tests
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run all tests
|
||||||
|
uv run pytest
|
||||||
|
|
||||||
|
# Run specific test files
|
||||||
|
uv run pytest tests/test_base_collector.py
|
||||||
|
|
||||||
|
# Run with coverage
|
||||||
|
uv run pytest --cov=data --cov-report=html
|
||||||
|
|
||||||
|
# Run integration tests
|
||||||
|
uv run pytest tests/integration/
|
||||||
|
```
|
||||||
|
|
||||||
|
### Test Organization
|
||||||
|
|
||||||
|
- **Unit tests**: Individual component testing
|
||||||
|
- **Integration tests**: Cross-component functionality
|
||||||
|
- **Performance tests**: Load and stress testing
|
||||||
|
- **End-to-end tests**: Full system workflows
|
||||||
|
|
||||||
|
### Writing Tests
|
||||||
|
|
||||||
|
Follow these patterns when writing tests:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import pytest
|
||||||
|
import asyncio
|
||||||
|
from data.exchanges import create_okx_collector
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_okx_collector():
|
||||||
|
collector = create_okx_collector('BTC-USDT')
|
||||||
|
assert collector is not None
|
||||||
|
|
||||||
|
# Test lifecycle
|
||||||
|
await collector.start()
|
||||||
|
status = collector.get_status()
|
||||||
|
assert status['status'] == 'running'
|
||||||
|
|
||||||
|
await collector.stop()
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🚀 Deployment
|
||||||
|
|
||||||
|
### Development Deployment
|
||||||
|
|
||||||
|
For local development:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Start services
|
||||||
|
docker-compose up -d
|
||||||
|
|
||||||
|
# Initialize database
|
||||||
|
uv run python scripts/init_database.py
|
||||||
|
|
||||||
|
# Start data collection
|
||||||
|
uv run python scripts/start_collectors.py
|
||||||
|
```
|
||||||
|
|
||||||
|
### Production Deployment
|
||||||
|
|
||||||
|
For production environments:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Use production docker-compose
|
||||||
|
docker-compose -f docker-compose.prod.yml up -d
|
||||||
|
|
||||||
|
# Set production environment
|
||||||
|
export ENV=production
|
||||||
|
export LOG_LEVEL=INFO
|
||||||
|
|
||||||
|
# Start with monitoring
|
||||||
|
uv run python scripts/production_start.py
|
||||||
|
```
|
||||||
|
|
||||||
|
### Docker Deployment
|
||||||
|
|
||||||
|
Using Docker containers:
|
||||||
|
|
||||||
|
```dockerfile
|
||||||
|
FROM python:3.11-slim
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
COPY requirements.txt .
|
||||||
|
RUN pip install -r requirements.txt
|
||||||
|
|
||||||
|
COPY . .
|
||||||
|
CMD ["python", "-m", "scripts.production_start"]
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🔗 Related Documentation
|
||||||
|
|
||||||
|
- **[Components Documentation](../components/)** - Technical component details
|
||||||
|
- **[Architecture Overview](../architecture/)** - System design
|
||||||
|
- **[Exchange Documentation](../exchanges/)** - Exchange integrations
|
||||||
|
- **[API Reference](../reference/)** - Technical specifications
|
||||||
|
|
||||||
|
## 📞 Support & Troubleshooting
|
||||||
|
|
||||||
|
### Common Issues
|
||||||
|
|
||||||
|
1. **Database Connection Errors**
|
||||||
|
- Check Docker services: `docker-compose ps`
|
||||||
|
- Verify environment variables in `.env`
|
||||||
|
- Test connection: `uv run python scripts/test_db_connection.py`
|
||||||
|
|
||||||
|
2. **Collector Failures**
|
||||||
|
- Check logs: `tail -f logs/collector_error.log`
|
||||||
|
- Verify configuration: Review `config/*.json` files
|
||||||
|
- Test manually: `uv run python scripts/test_okx_collector.py`
|
||||||
|
|
||||||
|
3. **Performance Issues**
|
||||||
|
- Monitor resource usage: `docker stats`
|
||||||
|
- Check message rates: Collector status endpoints
|
||||||
|
- Optimize configuration: Adjust health check intervals
|
||||||
|
|
||||||
|
### Getting Help
|
||||||
|
|
||||||
|
1. **Check Documentation**: Review relevant section documentation
|
||||||
|
2. **Review Logs**: System logs in `./logs/` directory
|
||||||
|
3. **Test Components**: Use built-in test scripts
|
||||||
|
4. **Check Status**: Use status and health check methods
|
||||||
|
|
||||||
|
### Debug Mode
|
||||||
|
|
||||||
|
Enable detailed debugging:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export LOG_LEVEL=DEBUG
|
||||||
|
uv run python your_script.py
|
||||||
|
|
||||||
|
# Check detailed logs
|
||||||
|
tail -f logs/*_debug.log
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*For the complete documentation index, see the [main documentation README](../README.md).*
|
||||||
390
docs/reference/README.md
Normal file
390
docs/reference/README.md
Normal file
@ -0,0 +1,390 @@
|
|||||||
|
# Reference Documentation
|
||||||
|
|
||||||
|
This section contains technical specifications, API references, and detailed documentation for the TCP Dashboard platform.
|
||||||
|
|
||||||
|
## 📋 Contents
|
||||||
|
|
||||||
|
### Technical Specifications
|
||||||
|
|
||||||
|
- **[Project Specification](specification.md)** - *Technical specifications and requirements*
|
||||||
|
- System requirements and constraints
|
||||||
|
- Database schema specifications
|
||||||
|
- API endpoint definitions
|
||||||
|
- Data format specifications
|
||||||
|
- Integration requirements
|
||||||
|
|
||||||
|
### API References
|
||||||
|
|
||||||
|
#### Data Collection APIs
|
||||||
|
|
||||||
|
```python
|
||||||
|
# BaseDataCollector API
|
||||||
|
class BaseDataCollector:
|
||||||
|
async def start() -> bool
|
||||||
|
async def stop(force: bool = False) -> None
|
||||||
|
async def restart() -> bool
|
||||||
|
def get_status() -> Dict[str, Any]
|
||||||
|
def get_health_status() -> Dict[str, Any]
|
||||||
|
def add_data_callback(data_type: DataType, callback: Callable) -> None
|
||||||
|
|
||||||
|
# CollectorManager API
|
||||||
|
class CollectorManager:
|
||||||
|
def add_collector(collector: BaseDataCollector) -> None
|
||||||
|
async def start() -> bool
|
||||||
|
async def stop() -> None
|
||||||
|
def get_status() -> Dict[str, Any]
|
||||||
|
def list_collectors() -> List[str]
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Exchange Factory APIs
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Factory Pattern API
|
||||||
|
class ExchangeFactory:
|
||||||
|
@staticmethod
|
||||||
|
def create_collector(config: ExchangeCollectorConfig) -> BaseDataCollector
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def create_multiple_collectors(configs: List[ExchangeCollectorConfig]) -> List[BaseDataCollector]
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def get_supported_exchanges() -> List[str]
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def validate_config(config: ExchangeCollectorConfig) -> bool
|
||||||
|
|
||||||
|
# Configuration API
|
||||||
|
@dataclass
|
||||||
|
class ExchangeCollectorConfig:
|
||||||
|
exchange: str
|
||||||
|
symbol: str
|
||||||
|
data_types: List[DataType]
|
||||||
|
auto_restart: bool = True
|
||||||
|
health_check_interval: float = 30.0
|
||||||
|
store_raw_data: bool = True
|
||||||
|
custom_params: Optional[Dict[str, Any]] = None
|
||||||
|
```
|
||||||
|
|
||||||
|
## 📊 Data Schemas
|
||||||
|
|
||||||
|
### Market Data Point
|
||||||
|
|
||||||
|
The standardized data structure for all market data:
|
||||||
|
|
||||||
|
```python
|
||||||
|
@dataclass
|
||||||
|
class MarketDataPoint:
|
||||||
|
exchange: str # Exchange name (e.g., 'okx', 'binance')
|
||||||
|
symbol: str # Trading symbol (e.g., 'BTC-USDT')
|
||||||
|
timestamp: datetime # Data timestamp (UTC)
|
||||||
|
data_type: DataType # Type of data (TRADE, ORDERBOOK, etc.)
|
||||||
|
data: Dict[str, Any] # Raw data payload
|
||||||
|
```
|
||||||
|
|
||||||
|
### Data Types
|
||||||
|
|
||||||
|
```python
|
||||||
|
class DataType(Enum):
|
||||||
|
TICKER = "ticker" # Price and volume updates
|
||||||
|
TRADE = "trade" # Individual trade executions
|
||||||
|
ORDERBOOK = "orderbook" # Order book snapshots
|
||||||
|
CANDLE = "candle" # OHLCV candle data
|
||||||
|
BALANCE = "balance" # Account balance updates
|
||||||
|
```
|
||||||
|
|
||||||
|
### Status Schemas
|
||||||
|
|
||||||
|
#### Collector Status
|
||||||
|
|
||||||
|
```python
|
||||||
|
{
|
||||||
|
'exchange': str, # Exchange name
|
||||||
|
'status': str, # Current status (running, stopped, error)
|
||||||
|
'should_be_running': bool, # Desired state
|
||||||
|
'symbols': List[str], # Configured symbols
|
||||||
|
'data_types': List[str], # Data types being collected
|
||||||
|
'auto_restart': bool, # Auto-restart enabled
|
||||||
|
'health': {
|
||||||
|
'time_since_heartbeat': float, # Seconds since last heartbeat
|
||||||
|
'time_since_data': float, # Seconds since last data
|
||||||
|
'max_silence_duration': float # Max allowed silence
|
||||||
|
},
|
||||||
|
'statistics': {
|
||||||
|
'messages_received': int, # Total messages received
|
||||||
|
'messages_processed': int, # Successfully processed
|
||||||
|
'errors': int, # Error count
|
||||||
|
'restarts': int, # Restart count
|
||||||
|
'uptime_seconds': float, # Current uptime
|
||||||
|
'reconnect_attempts': int, # Current reconnect attempts
|
||||||
|
'last_message_time': str, # ISO timestamp
|
||||||
|
'connection_uptime': str, # Connection start time
|
||||||
|
'last_error': str, # Last error message
|
||||||
|
'last_restart_time': str # Last restart time
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Health Status
|
||||||
|
|
||||||
|
```python
|
||||||
|
{
|
||||||
|
'is_healthy': bool, # Overall health status
|
||||||
|
'issues': List[str], # List of current issues
|
||||||
|
'status': str, # Current collector status
|
||||||
|
'last_heartbeat': str, # Last heartbeat timestamp
|
||||||
|
'last_data_received': str, # Last data timestamp
|
||||||
|
'should_be_running': bool, # Expected state
|
||||||
|
'is_running': bool # Actual running state
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🔧 Configuration Schemas
|
||||||
|
|
||||||
|
### Database Configuration
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"database": {
|
||||||
|
"url": "postgresql://user:pass@host:port/db",
|
||||||
|
"pool_size": 10,
|
||||||
|
"max_overflow": 20,
|
||||||
|
"pool_timeout": 30,
|
||||||
|
"pool_recycle": 3600
|
||||||
|
},
|
||||||
|
"tables": {
|
||||||
|
"market_data": "market_data",
|
||||||
|
"raw_trades": "raw_trades",
|
||||||
|
"collector_status": "collector_status"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Exchange Configuration
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"exchange": "okx",
|
||||||
|
"connection": {
|
||||||
|
"public_ws_url": "wss://ws.okx.com:8443/ws/v5/public",
|
||||||
|
"ping_interval": 25.0,
|
||||||
|
"pong_timeout": 10.0,
|
||||||
|
"max_reconnect_attempts": 5,
|
||||||
|
"reconnect_delay": 5.0
|
||||||
|
},
|
||||||
|
"data_collection": {
|
||||||
|
"store_raw_data": true,
|
||||||
|
"health_check_interval": 30.0,
|
||||||
|
"auto_restart": true,
|
||||||
|
"buffer_size": 1000
|
||||||
|
},
|
||||||
|
"trading_pairs": [
|
||||||
|
{
|
||||||
|
"symbol": "BTC-USDT",
|
||||||
|
"enabled": true,
|
||||||
|
"data_types": ["trade", "orderbook"],
|
||||||
|
"channels": {
|
||||||
|
"trades": "trades",
|
||||||
|
"orderbook": "books5",
|
||||||
|
"ticker": "tickers"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Logging Configuration
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"logging": {
|
||||||
|
"level": "INFO",
|
||||||
|
"format": "detailed",
|
||||||
|
"console_output": true,
|
||||||
|
"file_output": true,
|
||||||
|
"cleanup": true,
|
||||||
|
"max_files": 30,
|
||||||
|
"log_directory": "./logs"
|
||||||
|
},
|
||||||
|
"components": {
|
||||||
|
"data_collectors": {
|
||||||
|
"level": "INFO",
|
||||||
|
"verbose": false
|
||||||
|
},
|
||||||
|
"websocket_clients": {
|
||||||
|
"level": "DEBUG",
|
||||||
|
"verbose": true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🌐 Protocol Specifications
|
||||||
|
|
||||||
|
### WebSocket Message Formats
|
||||||
|
|
||||||
|
#### OKX Message Format
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"arg": {
|
||||||
|
"channel": "trades",
|
||||||
|
"instId": "BTC-USDT"
|
||||||
|
},
|
||||||
|
"data": [
|
||||||
|
{
|
||||||
|
"instId": "BTC-USDT",
|
||||||
|
"tradeId": "12345678",
|
||||||
|
"px": "50000.5",
|
||||||
|
"sz": "0.001",
|
||||||
|
"side": "buy",
|
||||||
|
"ts": "1697123456789"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Subscription Message Format
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"op": "subscribe",
|
||||||
|
"args": [
|
||||||
|
{
|
||||||
|
"channel": "trades",
|
||||||
|
"instId": "BTC-USDT"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"channel": "books5",
|
||||||
|
"instId": "BTC-USDT"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Database Schemas
|
||||||
|
|
||||||
|
#### Market Data Table
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE market_data (
|
||||||
|
id SERIAL PRIMARY KEY,
|
||||||
|
exchange VARCHAR(50) NOT NULL,
|
||||||
|
symbol VARCHAR(50) NOT NULL,
|
||||||
|
data_type VARCHAR(20) NOT NULL,
|
||||||
|
timestamp TIMESTAMP WITH TIME ZONE NOT NULL,
|
||||||
|
data JSONB NOT NULL,
|
||||||
|
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||||
|
|
||||||
|
INDEX(exchange, symbol, timestamp),
|
||||||
|
INDEX(data_type, timestamp)
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Raw Trades Table
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE raw_trades (
|
||||||
|
id SERIAL PRIMARY KEY,
|
||||||
|
exchange VARCHAR(50) NOT NULL,
|
||||||
|
symbol VARCHAR(50) NOT NULL,
|
||||||
|
trade_id VARCHAR(100),
|
||||||
|
price DECIMAL(20, 8) NOT NULL,
|
||||||
|
size DECIMAL(20, 8) NOT NULL,
|
||||||
|
side VARCHAR(10) NOT NULL,
|
||||||
|
timestamp TIMESTAMP WITH TIME ZONE NOT NULL,
|
||||||
|
raw_data JSONB,
|
||||||
|
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||||
|
|
||||||
|
UNIQUE(exchange, symbol, trade_id),
|
||||||
|
INDEX(exchange, symbol, timestamp),
|
||||||
|
INDEX(timestamp)
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
## 📈 Performance Specifications
|
||||||
|
|
||||||
|
### System Requirements
|
||||||
|
|
||||||
|
#### Minimum Requirements
|
||||||
|
|
||||||
|
- **CPU**: 2 cores, 2.0 GHz
|
||||||
|
- **Memory**: 4 GB RAM
|
||||||
|
- **Storage**: 20 GB available space
|
||||||
|
- **Network**: Stable internet connection (100 Mbps+)
|
||||||
|
|
||||||
|
#### Recommended Requirements
|
||||||
|
|
||||||
|
- **CPU**: 4+ cores, 3.0+ GHz
|
||||||
|
- **Memory**: 8+ GB RAM
|
||||||
|
- **Storage**: 100+ GB SSD
|
||||||
|
- **Network**: High-speed internet (1 Gbps+)
|
||||||
|
|
||||||
|
### Performance Targets
|
||||||
|
|
||||||
|
#### Data Collection
|
||||||
|
|
||||||
|
- **Latency**: < 100ms from exchange to processing
|
||||||
|
- **Throughput**: 1000+ messages/second per collector
|
||||||
|
- **Uptime**: 99.9% availability
|
||||||
|
- **Memory Usage**: < 50 MB per collector
|
||||||
|
|
||||||
|
#### Database Operations
|
||||||
|
|
||||||
|
- **Insert Rate**: 10,000+ inserts/second
|
||||||
|
- **Query Response**: < 100ms for typical queries
|
||||||
|
- **Storage Growth**: ~1 GB/month per active trading pair
|
||||||
|
- **Retention**: 2+ years of historical data
|
||||||
|
|
||||||
|
## 🔒 Security Specifications
|
||||||
|
|
||||||
|
### Authentication & Authorization
|
||||||
|
|
||||||
|
- **API Keys**: Secure storage in environment variables
|
||||||
|
- **Database Access**: Connection pooling with authentication
|
||||||
|
- **WebSocket Connections**: TLS encryption for all connections
|
||||||
|
- **Logging**: No sensitive data in logs
|
||||||
|
|
||||||
|
### Data Protection
|
||||||
|
|
||||||
|
- **Encryption**: TLS 1.3 for all external communications
|
||||||
|
- **Data Validation**: Comprehensive input validation
|
||||||
|
- **Error Handling**: Secure error messages without data leakage
|
||||||
|
- **Backup**: Regular automated backups with encryption
|
||||||
|
|
||||||
|
## 🔗 Related Documentation
|
||||||
|
|
||||||
|
- **[Components Documentation](../components/)** - Implementation details
|
||||||
|
- **[Architecture Overview](../architecture/)** - System design
|
||||||
|
- **[Exchange Documentation](../exchanges/)** - Exchange integrations
|
||||||
|
- **[Setup Guide](../guides/)** - Configuration and deployment
|
||||||
|
|
||||||
|
## 📞 Support
|
||||||
|
|
||||||
|
### API Support
|
||||||
|
|
||||||
|
For API-related questions:
|
||||||
|
|
||||||
|
1. **Check Examples**: Review code examples in each API section
|
||||||
|
2. **Test Endpoints**: Use provided test scripts
|
||||||
|
3. **Validate Schemas**: Ensure data matches specified formats
|
||||||
|
4. **Review Logs**: Check detailed logs for API interactions
|
||||||
|
|
||||||
|
### Schema Validation
|
||||||
|
|
||||||
|
For data schema issues:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Validate data point structure
|
||||||
|
def validate_market_data_point(data_point):
|
||||||
|
required_fields = ['exchange', 'symbol', 'timestamp', 'data_type', 'data']
|
||||||
|
for field in required_fields:
|
||||||
|
if not hasattr(data_point, field):
|
||||||
|
raise ValueError(f"Missing required field: {field}")
|
||||||
|
|
||||||
|
if not isinstance(data_point.data_type, DataType):
|
||||||
|
raise ValueError("Invalid data_type")
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*For the complete documentation index, see the [main documentation README](../README.md).*
|
||||||
Loading…
x
Reference in New Issue
Block a user