documentation
This commit is contained in:
Vasily.onl
parent
4510181b39
commit
fa63e7eb2e
114
docs/README.md
114
docs/README.md
@ -2,59 +2,62 @@
|
||||
|
||||
Welcome to the **TCP Dashboard** (Trading Crypto Platform) documentation. This platform provides a comprehensive solution for cryptocurrency trading bot development, backtesting, and portfolio management.
|
||||
|
||||
## 📚 Documentation Index
|
||||
## 📚 Documentation Structure
|
||||
|
||||
### 🏗️ **Architecture & Design**
|
||||
The documentation is organized into specialized sections for better navigation and maintenance:
|
||||
|
||||
- **[Architecture Overview](architecture.md)** - High-level system architecture and component design
|
||||
- **[Project Specification](specification.md)** - Technical specifications and requirements
|
||||
- **[Crypto Bot PRD](crypto-bot-prd.md)** - Product Requirements Document for the crypto trading bot platform
|
||||
### 🏗️ **[Architecture & Design](architecture/)**
|
||||
|
||||
### 🚀 **Setup & Installation**
|
||||
- **[Architecture Overview](architecture/architecture.md)** - High-level system architecture and component design
|
||||
- **[Crypto Bot PRD](architecture/crypto-bot-prd.md)** - Product Requirements Document for the crypto trading bot platform
|
||||
|
||||
- **[Setup Guide](setup.md)** - Comprehensive setup instructions for new machines and environments
|
||||
- Environment configuration
|
||||
- Database setup with Docker
|
||||
- Development workflow
|
||||
- Production deployment
|
||||
### 🔧 **[Core Components](components/)**
|
||||
|
||||
### 🔧 **Core Systems**
|
||||
- **[Data Collectors](components/data_collectors.md)** - *Comprehensive guide to the enhanced data collector system*
|
||||
- BaseDataCollector abstract class with health monitoring
|
||||
- CollectorManager for centralized management
|
||||
- Exchange Factory Pattern for standardized collector creation
|
||||
- Modular Exchange Architecture for scalable implementation
|
||||
- Auto-restart and failure recovery mechanisms
|
||||
|
||||
#### Data Collection System
|
||||
- **[Logging System](components/logging.md)** - *Unified logging framework*
|
||||
- Multi-level logging with automatic cleanup
|
||||
- Console and file output with formatting
|
||||
- Performance monitoring integration
|
||||
|
||||
- **[Data Collectors Documentation](data_collectors.md)** - *Comprehensive guide to the enhanced data collector system*
|
||||
- **BaseDataCollector** abstract class with health monitoring
|
||||
- **CollectorManager** for centralized management
|
||||
- **Exchange Factory Pattern** for standardized collector creation
|
||||
- **Modular Exchange Architecture** for scalable implementation
|
||||
- Auto-restart and failure recovery
|
||||
- Health monitoring and alerting
|
||||
- Performance optimization
|
||||
- Integration examples
|
||||
- Troubleshooting guide
|
||||
### 🌐 **[Exchange Integrations](exchanges/)**
|
||||
|
||||
- **[OKX Collector Documentation](okx_collector.md)** - *Complete guide to OKX exchange integration*
|
||||
- **[OKX Collector](exchanges/okx_collector.md)** - *Complete guide to OKX exchange integration*
|
||||
- Real-time trades, orderbook, and ticker data collection
|
||||
- WebSocket connection management with OKX-specific ping/pong
|
||||
- Factory pattern usage and configuration
|
||||
- Data processing and validation
|
||||
- Monitoring and troubleshooting
|
||||
- Production deployment guide
|
||||
|
||||
#### Logging System
|
||||
- **[Exchange Overview](exchanges/)** - Multi-exchange architecture and comparison
|
||||
|
||||
- **[Enhanced Logging System](logging.md)** - Unified logging framework
|
||||
- Multi-level logging with automatic cleanup
|
||||
- Console and file output with formatting
|
||||
- Performance monitoring
|
||||
- Integration across all components
|
||||
### 📖 **[Setup & Guides](guides/)**
|
||||
|
||||
- **[Setup Guide](guides/setup.md)** - *Comprehensive setup instructions*
|
||||
- Environment configuration and prerequisites
|
||||
- Database setup with Docker and PostgreSQL
|
||||
- Development workflow and best practices
|
||||
- Production deployment guidelines
|
||||
|
||||
### 📋 **[Technical Reference](reference/)**
|
||||
|
||||
- **[Project Specification](reference/specification.md)** - *Technical specifications and requirements*
|
||||
- System requirements and constraints
|
||||
- Database schema specifications
|
||||
- API endpoint definitions
|
||||
- Data format specifications
|
||||
|
||||
## 🎯 **Quick Start**
|
||||
|
||||
1. **New to the platform?** Start with the [Setup Guide](setup.md)
|
||||
2. **Implementing data collectors?** See [Data Collectors Documentation](data_collectors.md)
|
||||
3. **Understanding the architecture?** Read [Architecture Overview](architecture.md)
|
||||
4. **Troubleshooting?** Check component-specific documentation
|
||||
1. **New to the platform?** Start with the [Setup Guide](guides/setup.md)
|
||||
2. **Implementing data collectors?** See [Data Collectors Documentation](components/data_collectors.md)
|
||||
3. **Understanding the architecture?** Read [Architecture Overview](architecture/architecture.md)
|
||||
4. **Exchange integration?** Check [Exchange Documentation](exchanges/)
|
||||
5. **Troubleshooting?** Check component-specific documentation
|
||||
|
||||
## 🏛️ **System Components**
|
||||
|
||||
@ -219,21 +222,38 @@ uv run pytest --cov=data --cov-report=html
|
||||
|
||||
---
|
||||
|
||||
## 📁 **File Structure**
|
||||
## 📁 **Documentation File Structure**
|
||||
|
||||
```
|
||||
TCPDashboard/
|
||||
├── docs/ # Documentation (you are here)
|
||||
├── data/ # Data collection system
|
||||
├── database/ # Database models and utilities
|
||||
├── utils/ # Shared utilities (logging, etc.)
|
||||
├── tests/ # Test suite
|
||||
├── examples/ # Usage examples
|
||||
├── config/ # Configuration files
|
||||
├── logs/ # Application logs
|
||||
└── scripts/ # Utility scripts
|
||||
docs/
|
||||
├── README.md # This file - main documentation index
|
||||
├── architecture/ # System architecture and design
|
||||
│ ├── README.md # Architecture overview
|
||||
│ ├── architecture.md # Technical architecture
|
||||
│ └── crypto-bot-prd.md # Product requirements
|
||||
├── components/ # Core system components
|
||||
│ ├── README.md # Component overview
|
||||
│ ├── data_collectors.md # Data collection system
|
||||
│ └── logging.md # Logging framework
|
||||
├── exchanges/ # Exchange integrations
|
||||
│ ├── README.md # Exchange overview
|
||||
│ └── okx_collector.md # OKX implementation
|
||||
├── guides/ # User guides and tutorials
|
||||
│ ├── README.md # Guide overview
|
||||
│ └── setup.md # Setup instructions
|
||||
└── reference/ # Technical reference
|
||||
├── README.md # Reference overview
|
||||
└── specification.md # Technical specifications
|
||||
```
|
||||
|
||||
## 🔗 **Navigation**
|
||||
|
||||
- **🏗️ [Architecture & Design](architecture/)** - System design and requirements
|
||||
- **🔧 [Core Components](components/)** - Technical implementation details
|
||||
- **🌐 [Exchange Integrations](exchanges/)** - Exchange-specific documentation
|
||||
- **📖 [Setup & Guides](guides/)** - User guides and tutorials
|
||||
- **📋 [Technical Reference](reference/)** - API specifications and schemas
|
||||
|
||||
---
|
||||
|
||||
*Last updated: $(date)*
|
||||
|
||||
41
docs/architecture/README.md
Normal file
41
docs/architecture/README.md
Normal file
@ -0,0 +1,41 @@
|
||||
# Architecture Documentation
|
||||
|
||||
This section contains system architecture and design documentation for the TCP Dashboard platform.
|
||||
|
||||
## 📋 Contents
|
||||
|
||||
### System Architecture
|
||||
|
||||
- **[Architecture Overview](architecture.md)** - *High-level system architecture and component design*
|
||||
- Core system components and interactions
|
||||
- Data flow and processing pipelines
|
||||
- Service architecture and deployment patterns
|
||||
- Technology stack and infrastructure
|
||||
|
||||
### Product Requirements
|
||||
|
||||
- **[Crypto Bot PRD](crypto-bot-prd.md)** - *Product Requirements Document for the crypto trading bot platform*
|
||||
- Platform vision and objectives
|
||||
- Feature specifications and requirements
|
||||
- User personas and use cases
|
||||
- Technical requirements and constraints
|
||||
- Implementation roadmap and milestones
|
||||
|
||||
## 🏗️ System Overview
|
||||
|
||||
The TCP Dashboard follows a modular, microservices-inspired architecture designed for:
|
||||
|
||||
- **Scalability**: Horizontal scaling of individual components
|
||||
- **Reliability**: Fault tolerance and auto-recovery mechanisms
|
||||
- **Maintainability**: Clear separation of concerns and modular design
|
||||
- **Extensibility**: Easy addition of new exchanges, strategies, and features
|
||||
|
||||
## 🔗 Related Documentation
|
||||
|
||||
- **[Components Documentation](../components/)** - Technical implementation details
|
||||
- **[Setup Guide](../guides/setup.md)** - System setup and configuration
|
||||
- **[Reference Documentation](../reference/)** - API specifications and technical references
|
||||
|
||||
---
|
||||
|
||||
*For the complete documentation index, see the [main documentation README](../README.md).*
|
||||
135
docs/components/README.md
Normal file
135
docs/components/README.md
Normal file
@ -0,0 +1,135 @@
|
||||
# Components Documentation
|
||||
|
||||
This section contains detailed technical documentation for all system components in the TCP Dashboard platform.
|
||||
|
||||
## 📋 Contents
|
||||
|
||||
### Data Collection System
|
||||
|
||||
- **[Data Collectors](data_collectors.md)** - *Comprehensive guide to the enhanced data collector system*
|
||||
- **BaseDataCollector** abstract class with health monitoring
|
||||
- **CollectorManager** for centralized management
|
||||
- **Exchange Factory Pattern** for standardized collector creation
|
||||
- **Modular Exchange Architecture** for scalable implementation
|
||||
- Auto-restart and failure recovery mechanisms
|
||||
- Health monitoring and alerting systems
|
||||
- Performance optimization techniques
|
||||
- Integration examples and patterns
|
||||
- Comprehensive troubleshooting guide
|
||||
|
||||
### Logging & Monitoring
|
||||
|
||||
- **[Enhanced Logging System](logging.md)** - *Unified logging framework*
|
||||
- Multi-level logging with automatic cleanup
|
||||
- Console and file output with formatting
|
||||
- Performance monitoring integration
|
||||
- Cross-component logging standards
|
||||
- Log aggregation and analysis
|
||||
|
||||
## 🔧 Component Architecture
|
||||
|
||||
### Core Components
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ CollectorManager │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ Global Health Monitor │ │
|
||||
│ │ • System-wide health checks │ │
|
||||
│ │ • Auto-restart coordination │ │
|
||||
│ │ • Performance analytics │ │
|
||||
│ └─────────────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌─────────────────┐ ┌─────────────────┐ ┌──────────────┐ │
|
||||
│ │ OKX Collector │ │Binance Collector│ │ Custom │ │
|
||||
│ │ │ │ │ │ Collector │ │
|
||||
│ │ • Health Monitor│ │ • Health Monitor│ │ • Health Mon │ │
|
||||
│ │ • Auto-restart │ │ • Auto-restart │ │ • Auto-resta │ │
|
||||
│ │ • Data Validate │ │ • Data Validate │ │ • Data Valid │ │
|
||||
│ └─────────────────┘ └─────────────────┘ └──────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Design Patterns
|
||||
|
||||
- **Factory Pattern**: Standardized component creation across exchanges
|
||||
- **Observer Pattern**: Event-driven data processing and callbacks
|
||||
- **Strategy Pattern**: Pluggable data processing strategies
|
||||
- **Singleton Pattern**: Centralized logging and configuration management
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### Using Components
|
||||
|
||||
```python
|
||||
# Data Collector usage
|
||||
from data.exchanges import create_okx_collector
|
||||
from data.base_collector import DataType
|
||||
|
||||
collector = create_okx_collector(
|
||||
symbol='BTC-USDT',
|
||||
data_types=[DataType.TRADE, DataType.ORDERBOOK]
|
||||
)
|
||||
|
||||
# Logging usage
|
||||
from utils.logger import get_logger
|
||||
|
||||
logger = get_logger("my_component")
|
||||
logger.info("Component initialized")
|
||||
```
|
||||
|
||||
### Component Integration
|
||||
|
||||
```python
|
||||
# Integrating multiple components
|
||||
from data.collector_manager import CollectorManager
|
||||
from utils.logger import get_logger
|
||||
|
||||
manager = CollectorManager("production_system")
|
||||
logger = get_logger("system_manager")
|
||||
|
||||
# Components work together seamlessly
|
||||
await manager.start()
|
||||
logger.info("System started successfully")
|
||||
```
|
||||
|
||||
## 📊 Performance & Monitoring
|
||||
|
||||
### Health Monitoring
|
||||
|
||||
All components include built-in health monitoring:
|
||||
|
||||
- **Real-time Status**: Component state and performance metrics
|
||||
- **Auto-Recovery**: Automatic restart on failures
|
||||
- **Performance Tracking**: Message rates, uptime, error rates
|
||||
- **Alerting**: Configurable alerts for component health
|
||||
|
||||
### Logging Integration
|
||||
|
||||
Unified logging across all components:
|
||||
|
||||
- **Structured Logging**: JSON-formatted logs for analysis
|
||||
- **Multiple Levels**: Debug, Info, Warning, Error levels
|
||||
- **Automatic Cleanup**: Log rotation and old file cleanup
|
||||
- **Performance Metrics**: Built-in performance tracking
|
||||
|
||||
## 🔗 Related Documentation
|
||||
|
||||
- **[Exchange Documentation](../exchanges/)** - Exchange-specific implementations
|
||||
- **[Architecture Overview](../architecture/)** - System design and patterns
|
||||
- **[Setup Guide](../guides/setup.md)** - Component configuration and deployment
|
||||
- **[API Reference](../reference/)** - Technical specifications
|
||||
|
||||
## 📈 Future Components
|
||||
|
||||
Planned component additions:
|
||||
|
||||
- **Strategy Engine**: Trading strategy execution framework
|
||||
- **Portfolio Manager**: Position and risk management
|
||||
- **Dashboard UI**: Web-based monitoring and control interface
|
||||
- **Alert Manager**: Advanced alerting and notification system
|
||||
- **Data Analytics**: Historical data analysis and reporting
|
||||
|
||||
---
|
||||
|
||||
*For the complete documentation index, see the [main documentation README](../README.md).*
|
||||
297
docs/exchanges/README.md
Normal file
297
docs/exchanges/README.md
Normal file
@ -0,0 +1,297 @@
|
||||
# Exchange Documentation
|
||||
|
||||
This section contains detailed documentation for all cryptocurrency exchange integrations in the TCP Dashboard platform.
|
||||
|
||||
## 📋 Contents
|
||||
|
||||
### Supported Exchanges
|
||||
|
||||
#### Production Ready
|
||||
|
||||
- **[OKX Collector](okx_collector.md)** - *Complete guide to OKX exchange integration*
|
||||
- Real-time trades, orderbook, and ticker data collection
|
||||
- WebSocket connection management with OKX-specific ping/pong
|
||||
- Factory pattern usage and configuration
|
||||
- Data processing and validation
|
||||
- Monitoring and troubleshooting
|
||||
- Production deployment guide
|
||||
|
||||
#### Planned Integrations
|
||||
|
||||
- **Binance** - Major global exchange (development planned)
|
||||
- **Coinbase Pro** - US-regulated exchange (development planned)
|
||||
- **Kraken** - European exchange (development planned)
|
||||
- **Bybit** - Derivatives exchange (development planned)
|
||||
|
||||
## 🏗️ Exchange Architecture
|
||||
|
||||
### Modular Design
|
||||
|
||||
Each exchange implementation follows a standardized structure:
|
||||
|
||||
```
|
||||
data/exchanges/
|
||||
├── __init__.py # Main exports and factory
|
||||
├── registry.py # Exchange registry and capabilities
|
||||
├── factory.py # Factory pattern for collectors
|
||||
└── {exchange}/ # Exchange-specific implementation
|
||||
├── __init__.py # Exchange exports
|
||||
├── collector.py # {Exchange}Collector class
|
||||
└── websocket.py # {Exchange}WebSocketClient class
|
||||
```
|
||||
|
||||
### Standardized Interface
|
||||
|
||||
All exchange collectors implement the same interface:
|
||||
|
||||
```python
|
||||
from data.exchanges import ExchangeFactory, ExchangeCollectorConfig
|
||||
from data.base_collector import DataType
|
||||
|
||||
# Unified configuration across all exchanges
|
||||
config = ExchangeCollectorConfig(
|
||||
exchange='okx', # or 'binance', 'coinbase', etc.
|
||||
symbol='BTC-USDT',
|
||||
data_types=[DataType.TRADE, DataType.ORDERBOOK],
|
||||
auto_restart=True
|
||||
)
|
||||
|
||||
collector = ExchangeFactory.create_collector(config)
|
||||
```
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### Using Factory Pattern
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from data.exchanges import get_supported_exchanges, create_okx_collector
|
||||
from data.base_collector import DataType
|
||||
|
||||
async def main():
|
||||
# Check supported exchanges
|
||||
exchanges = get_supported_exchanges()
|
||||
print(f"Supported: {exchanges}") # ['okx']
|
||||
|
||||
# Create OKX collector
|
||||
collector = create_okx_collector(
|
||||
symbol='BTC-USDT',
|
||||
data_types=[DataType.TRADE, DataType.ORDERBOOK]
|
||||
)
|
||||
|
||||
# Add data callback
|
||||
def on_trade(data_point):
|
||||
print(f"Trade: {data_point.data}")
|
||||
|
||||
collector.add_data_callback(DataType.TRADE, on_trade)
|
||||
|
||||
# Start collection
|
||||
await collector.start()
|
||||
await asyncio.sleep(60)
|
||||
await collector.stop()
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Multi-Exchange Setup
|
||||
|
||||
```python
|
||||
from data.exchanges import ExchangeFactory, ExchangeCollectorConfig
|
||||
from data.collector_manager import CollectorManager
|
||||
|
||||
async def setup_multi_exchange():
|
||||
manager = CollectorManager("multi_exchange_system")
|
||||
|
||||
# Future: Multiple exchanges
|
||||
configs = [
|
||||
ExchangeCollectorConfig('okx', 'BTC-USDT', [DataType.TRADE]),
|
||||
# ExchangeCollectorConfig('binance', 'BTC-USDT', [DataType.TRADE]),
|
||||
# ExchangeCollectorConfig('coinbase', 'BTC-USD', [DataType.TRADE])
|
||||
]
|
||||
|
||||
for config in configs:
|
||||
collector = ExchangeFactory.create_collector(config)
|
||||
manager.add_collector(collector)
|
||||
|
||||
await manager.start()
|
||||
return manager
|
||||
```
|
||||
|
||||
## 📊 Exchange Capabilities
|
||||
|
||||
### Data Types
|
||||
|
||||
Different exchanges support different data types:
|
||||
|
||||
| Exchange | Trades | Orderbook | Ticker | Candles | Balance |
|
||||
|----------|--------|-----------|--------|---------|---------|
|
||||
| OKX | ✅ | ✅ | ✅ | 🔄 | 🔄 |
|
||||
| Binance | 🔄 | 🔄 | 🔄 | 🔄 | 🔄 |
|
||||
| Coinbase | 🔄 | 🔄 | 🔄 | 🔄 | 🔄 |
|
||||
|
||||
Legend: ✅ Implemented, 🔄 Planned, ❌ Not supported
|
||||
|
||||
### Trading Pairs
|
||||
|
||||
Query supported trading pairs for each exchange:
|
||||
|
||||
```python
|
||||
from data.exchanges import ExchangeFactory
|
||||
|
||||
# Get supported pairs
|
||||
okx_pairs = ExchangeFactory.get_supported_pairs('okx')
|
||||
print(f"OKX pairs: {okx_pairs}")
|
||||
|
||||
# Get exchange information
|
||||
okx_info = ExchangeFactory.get_exchange_info('okx')
|
||||
print(f"OKX capabilities: {okx_info}")
|
||||
```
|
||||
|
||||
## 🔧 Exchange Configuration
|
||||
|
||||
### Common Configuration
|
||||
|
||||
All exchanges share common configuration options:
|
||||
|
||||
```python
|
||||
from data.exchanges import ExchangeCollectorConfig
|
||||
|
||||
config = ExchangeCollectorConfig(
|
||||
exchange='okx', # Exchange name
|
||||
symbol='BTC-USDT', # Trading pair
|
||||
data_types=[DataType.TRADE], # Data types to collect
|
||||
auto_restart=True, # Auto-restart on failures
|
||||
health_check_interval=30.0, # Health check interval
|
||||
store_raw_data=True, # Store raw exchange data
|
||||
custom_params={ # Exchange-specific parameters
|
||||
'ping_interval': 25.0,
|
||||
'max_reconnect_attempts': 5
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
### Exchange-Specific Configuration
|
||||
|
||||
Each exchange has specific configuration files:
|
||||
|
||||
- **OKX**: `config/okx_config.json`
|
||||
- **Binance**: `config/binance_config.json` (planned)
|
||||
- **Coinbase**: `config/coinbase_config.json` (planned)
|
||||
|
||||
## 📈 Performance Comparison
|
||||
|
||||
### Real-time Data Rates
|
||||
|
||||
Approximate message rates for different exchanges:
|
||||
|
||||
| Exchange | Trades/sec | Orderbook Updates/sec | Latency |
|
||||
|----------|------------|----------------------|---------|
|
||||
| OKX | 5-50 | 10-100 | ~50ms |
|
||||
| Binance | TBD | TBD | TBD |
|
||||
| Coinbase | TBD | TBD | TBD |
|
||||
|
||||
*Note: Rates vary by trading pair activity*
|
||||
|
||||
### Resource Usage
|
||||
|
||||
Memory and CPU usage per collector:
|
||||
|
||||
| Exchange | Memory (MB) | CPU (%) | Network (KB/s) |
|
||||
|----------|-------------|---------|----------------|
|
||||
| OKX | 15-25 | 1-3 | 5-20 |
|
||||
| Binance | TBD | TBD | TBD |
|
||||
| Coinbase | TBD | TBD | TBD |
|
||||
|
||||
## 🔍 Monitoring & Debugging
|
||||
|
||||
### Exchange Status
|
||||
|
||||
Monitor exchange-specific metrics:
|
||||
|
||||
```python
|
||||
# Get exchange status
|
||||
status = collector.get_status()
|
||||
print(f"Exchange: {status['exchange']}")
|
||||
print(f"WebSocket State: {status['websocket_state']}")
|
||||
print(f"Messages Processed: {status['messages_processed']}")
|
||||
|
||||
# Exchange-specific metrics
|
||||
if 'websocket_stats' in status:
|
||||
ws_stats = status['websocket_stats']
|
||||
print(f"Reconnections: {ws_stats['reconnections']}")
|
||||
print(f"Ping/Pong: {ws_stats['pings_sent']}/{ws_stats['pongs_received']}")
|
||||
```
|
||||
|
||||
### Debug Mode
|
||||
|
||||
Enable exchange-specific debugging:
|
||||
|
||||
```python
|
||||
import os
|
||||
os.environ['LOG_LEVEL'] = 'DEBUG'
|
||||
|
||||
# Detailed exchange logging
|
||||
collector = create_okx_collector('BTC-USDT', [DataType.TRADE])
|
||||
# Check logs: ./logs/okx_collector_btc_usdt_debug.log
|
||||
```
|
||||
|
||||
## 🛠️ Adding New Exchanges
|
||||
|
||||
### Implementation Checklist
|
||||
|
||||
To add a new exchange:
|
||||
|
||||
1. **Create Exchange Folder**: `data/exchanges/{exchange}/`
|
||||
2. **Implement WebSocket Client**: `{exchange}/websocket.py`
|
||||
3. **Implement Collector**: `{exchange}/collector.py`
|
||||
4. **Add to Registry**: Update `registry.py`
|
||||
5. **Create Configuration**: `config/{exchange}_config.json`
|
||||
6. **Add Documentation**: `docs/exchanges/{exchange}_collector.md`
|
||||
7. **Add Tests**: `tests/test_{exchange}_collector.py`
|
||||
|
||||
### Implementation Template
|
||||
|
||||
```python
|
||||
# data/exchanges/newexchange/collector.py
|
||||
from data.base_collector import BaseDataCollector, DataType
|
||||
from .websocket import NewExchangeWebSocketClient
|
||||
|
||||
class NewExchangeCollector(BaseDataCollector):
|
||||
def __init__(self, symbol: str, **kwargs):
|
||||
super().__init__("newexchange", [symbol], **kwargs)
|
||||
self.ws_client = NewExchangeWebSocketClient()
|
||||
|
||||
async def connect(self) -> bool:
|
||||
return await self.ws_client.connect()
|
||||
|
||||
# Implement other required methods...
|
||||
```
|
||||
|
||||
## 🔗 Related Documentation
|
||||
|
||||
- **[Components Documentation](../components/)** - Core system components
|
||||
- **[Architecture Overview](../architecture/)** - System design
|
||||
- **[Setup Guide](../guides/setup.md)** - Configuration and deployment
|
||||
- **[API Reference](../reference/)** - Technical specifications
|
||||
|
||||
## 📞 Support
|
||||
|
||||
### Exchange-Specific Issues
|
||||
|
||||
For exchange-specific problems:
|
||||
|
||||
1. **Check Status**: Use `get_status()` and `get_health_status()`
|
||||
2. **Review Logs**: Check exchange-specific log files
|
||||
3. **Verify Configuration**: Confirm exchange configuration files
|
||||
4. **Test Connection**: Run exchange-specific test scripts
|
||||
|
||||
### Common Issues
|
||||
|
||||
- **Rate Limiting**: Each exchange has different rate limits
|
||||
- **Symbol Formats**: Trading pair naming conventions vary
|
||||
- **WebSocket Protocols**: Each exchange has unique WebSocket requirements
|
||||
- **Data Formats**: Message structures differ between exchanges
|
||||
|
||||
---
|
||||
|
||||
*For the complete documentation index, see the [main documentation README](../README.md).*
|
||||
309
docs/guides/README.md
Normal file
309
docs/guides/README.md
Normal file
@ -0,0 +1,309 @@
|
||||
# Guides Documentation
|
||||
|
||||
This section contains user guides, tutorials, and setup instructions for the TCP Dashboard platform.
|
||||
|
||||
## 📋 Contents
|
||||
|
||||
### Setup & Installation
|
||||
|
||||
- **[Setup Guide](setup.md)** - *Comprehensive setup instructions for new machines and environments*
|
||||
- Environment configuration and prerequisites
|
||||
- Database setup with Docker and PostgreSQL
|
||||
- Development workflow and best practices
|
||||
- Production deployment guidelines
|
||||
- Troubleshooting common setup issues
|
||||
|
||||
### Quick Start Guides
|
||||
|
||||
#### For Developers
|
||||
|
||||
```bash
|
||||
# Quick setup for development
|
||||
git clone <repository>
|
||||
cd TCPDashboard
|
||||
uv sync
|
||||
cp .env.example .env
|
||||
docker-compose up -d
|
||||
uv run python scripts/init_database.py
|
||||
```
|
||||
|
||||
#### For Users
|
||||
|
||||
```python
|
||||
# Quick data collection setup
|
||||
from data.exchanges import create_okx_collector
|
||||
from data.base_collector import DataType
|
||||
|
||||
collector = create_okx_collector(
|
||||
symbol='BTC-USDT',
|
||||
data_types=[DataType.TRADE]
|
||||
)
|
||||
await collector.start()
|
||||
```
|
||||
|
||||
## 🚀 Tutorial Series
|
||||
|
||||
### Getting Started
|
||||
|
||||
1. **[Environment Setup](setup.md#environment-setup)** - Setting up your development environment
|
||||
2. **[First Data Collector](setup.md#first-collector)** - Creating your first data collector
|
||||
3. **[Database Integration](setup.md#database-setup)** - Connecting to the database
|
||||
4. **[Adding Monitoring](setup.md#monitoring)** - Setting up logging and monitoring
|
||||
|
||||
### Advanced Topics
|
||||
|
||||
1. **[Multi-Exchange Setup](setup.md#multi-exchange)** - Collecting from multiple exchanges
|
||||
2. **[Production Deployment](setup.md#production)** - Deploying to production
|
||||
3. **[Performance Optimization](setup.md#optimization)** - Optimizing for high throughput
|
||||
4. **[Custom Integrations](setup.md#custom)** - Building custom data sources
|
||||
|
||||
## 🛠️ Development Workflow
|
||||
|
||||
### Daily Development
|
||||
|
||||
```bash
|
||||
# Start development environment
|
||||
docker-compose up -d
|
||||
|
||||
# Install new dependencies
|
||||
uv add package-name
|
||||
|
||||
# Run tests
|
||||
uv run pytest
|
||||
|
||||
# Check code quality
|
||||
uv run black .
|
||||
uv run isort .
|
||||
```
|
||||
|
||||
### Code Organization
|
||||
|
||||
- **`data/`**: Data collection and processing
|
||||
- **`database/`**: Database models and utilities
|
||||
- **`utils/`**: Shared utilities and logging
|
||||
- **`tests/`**: Test suite
|
||||
- **`docs/`**: Documentation
|
||||
- **`config/`**: Configuration files
|
||||
|
||||
### Best Practices
|
||||
|
||||
1. **Follow existing patterns**: Use established code patterns
|
||||
2. **Write tests first**: TDD approach for new features
|
||||
3. **Document changes**: Update docs with code changes
|
||||
4. **Use type hints**: Full type annotation coverage
|
||||
5. **Handle errors**: Robust error handling throughout
|
||||
|
||||
## 🔧 Configuration Management
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Key environment variables to configure:
|
||||
|
||||
```bash
|
||||
# Database
|
||||
DATABASE_URL=postgresql://user:pass@localhost:5432/tcp_dashboard
|
||||
|
||||
# Logging
|
||||
LOG_LEVEL=INFO
|
||||
LOG_CLEANUP=true
|
||||
|
||||
# Data Collection
|
||||
DEFAULT_HEALTH_CHECK_INTERVAL=30
|
||||
AUTO_RESTART=true
|
||||
```
|
||||
|
||||
### Configuration Files
|
||||
|
||||
The platform uses JSON configuration files:
|
||||
|
||||
- **`config/okx_config.json`**: OKX exchange settings
|
||||
- **`config/database_config.json`**: Database configuration
|
||||
- **`config/logging_config.json`**: Logging settings
|
||||
|
||||
### Security Best Practices
|
||||
|
||||
- **Never commit secrets**: Use `.env` files for sensitive data
|
||||
- **Validate inputs**: Comprehensive input validation
|
||||
- **Use HTTPS**: Secure connections in production
|
||||
- **Regular updates**: Keep dependencies updated
|
||||
|
||||
## 📊 Monitoring & Observability
|
||||
|
||||
### Health Monitoring
|
||||
|
||||
The platform includes comprehensive health monitoring:
|
||||
|
||||
```python
|
||||
# Check system health
|
||||
from data.collector_manager import CollectorManager
|
||||
|
||||
manager = CollectorManager()
|
||||
status = manager.get_status()
|
||||
|
||||
print(f"Running collectors: {status['statistics']['running_collectors']}")
|
||||
print(f"Failed collectors: {status['statistics']['failed_collectors']}")
|
||||
```
|
||||
|
||||
### Logging
|
||||
|
||||
Structured logging across all components:
|
||||
|
||||
```python
|
||||
from utils.logger import get_logger
|
||||
|
||||
logger = get_logger("my_component")
|
||||
logger.info("Component started", extra={"component": "my_component"})
|
||||
```
|
||||
|
||||
### Performance Metrics
|
||||
|
||||
Built-in performance tracking:
|
||||
|
||||
- **Message rates**: Real-time data processing rates
|
||||
- **Error rates**: System health and stability
|
||||
- **Resource usage**: Memory and CPU utilization
|
||||
- **Uptime**: Component availability metrics
|
||||
|
||||
## 🧪 Testing
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
uv run pytest
|
||||
|
||||
# Run specific test files
|
||||
uv run pytest tests/test_base_collector.py
|
||||
|
||||
# Run with coverage
|
||||
uv run pytest --cov=data --cov-report=html
|
||||
|
||||
# Run integration tests
|
||||
uv run pytest tests/integration/
|
||||
```
|
||||
|
||||
### Test Organization
|
||||
|
||||
- **Unit tests**: Individual component testing
|
||||
- **Integration tests**: Cross-component functionality
|
||||
- **Performance tests**: Load and stress testing
|
||||
- **End-to-end tests**: Full system workflows
|
||||
|
||||
### Writing Tests
|
||||
|
||||
Follow these patterns when writing tests:
|
||||
|
||||
```python
|
||||
import pytest
|
||||
import asyncio
|
||||
from data.exchanges import create_okx_collector
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_okx_collector():
|
||||
collector = create_okx_collector('BTC-USDT')
|
||||
assert collector is not None
|
||||
|
||||
# Test lifecycle
|
||||
await collector.start()
|
||||
status = collector.get_status()
|
||||
assert status['status'] == 'running'
|
||||
|
||||
await collector.stop()
|
||||
```
|
||||
|
||||
## 🚀 Deployment
|
||||
|
||||
### Development Deployment
|
||||
|
||||
For local development:
|
||||
|
||||
```bash
|
||||
# Start services
|
||||
docker-compose up -d
|
||||
|
||||
# Initialize database
|
||||
uv run python scripts/init_database.py
|
||||
|
||||
# Start data collection
|
||||
uv run python scripts/start_collectors.py
|
||||
```
|
||||
|
||||
### Production Deployment
|
||||
|
||||
For production environments:
|
||||
|
||||
```bash
|
||||
# Use production docker-compose
|
||||
docker-compose -f docker-compose.prod.yml up -d
|
||||
|
||||
# Set production environment
|
||||
export ENV=production
|
||||
export LOG_LEVEL=INFO
|
||||
|
||||
# Start with monitoring
|
||||
uv run python scripts/production_start.py
|
||||
```
|
||||
|
||||
### Docker Deployment
|
||||
|
||||
Using Docker containers:
|
||||
|
||||
```dockerfile
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
COPY requirements.txt .
|
||||
RUN pip install -r requirements.txt
|
||||
|
||||
COPY . .
|
||||
CMD ["python", "-m", "scripts.production_start"]
|
||||
```
|
||||
|
||||
## 🔗 Related Documentation
|
||||
|
||||
- **[Components Documentation](../components/)** - Technical component details
|
||||
- **[Architecture Overview](../architecture/)** - System design
|
||||
- **[Exchange Documentation](../exchanges/)** - Exchange integrations
|
||||
- **[API Reference](../reference/)** - Technical specifications
|
||||
|
||||
## 📞 Support & Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Database Connection Errors**
|
||||
- Check Docker services: `docker-compose ps`
|
||||
- Verify environment variables in `.env`
|
||||
- Test connection: `uv run python scripts/test_db_connection.py`
|
||||
|
||||
2. **Collector Failures**
|
||||
- Check logs: `tail -f logs/collector_error.log`
|
||||
- Verify configuration: Review `config/*.json` files
|
||||
- Test manually: `uv run python scripts/test_okx_collector.py`
|
||||
|
||||
3. **Performance Issues**
|
||||
- Monitor resource usage: `docker stats`
|
||||
- Check message rates: Collector status endpoints
|
||||
- Optimize configuration: Adjust health check intervals
|
||||
|
||||
### Getting Help
|
||||
|
||||
1. **Check Documentation**: Review relevant section documentation
|
||||
2. **Review Logs**: System logs in `./logs/` directory
|
||||
3. **Test Components**: Use built-in test scripts
|
||||
4. **Check Status**: Use status and health check methods
|
||||
|
||||
### Debug Mode
|
||||
|
||||
Enable detailed debugging:
|
||||
|
||||
```bash
|
||||
export LOG_LEVEL=DEBUG
|
||||
uv run python your_script.py
|
||||
|
||||
# Check detailed logs
|
||||
tail -f logs/*_debug.log
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
*For the complete documentation index, see the [main documentation README](../README.md).*
|
||||
390
docs/reference/README.md
Normal file
390
docs/reference/README.md
Normal file
@ -0,0 +1,390 @@
|
||||
# Reference Documentation
|
||||
|
||||
This section contains technical specifications, API references, and detailed documentation for the TCP Dashboard platform.
|
||||
|
||||
## 📋 Contents
|
||||
|
||||
### Technical Specifications
|
||||
|
||||
- **[Project Specification](specification.md)** - *Technical specifications and requirements*
|
||||
- System requirements and constraints
|
||||
- Database schema specifications
|
||||
- API endpoint definitions
|
||||
- Data format specifications
|
||||
- Integration requirements
|
||||
|
||||
### API References
|
||||
|
||||
#### Data Collection APIs
|
||||
|
||||
```python
|
||||
# BaseDataCollector API
|
||||
class BaseDataCollector:
|
||||
async def start() -> bool
|
||||
async def stop(force: bool = False) -> None
|
||||
async def restart() -> bool
|
||||
def get_status() -> Dict[str, Any]
|
||||
def get_health_status() -> Dict[str, Any]
|
||||
def add_data_callback(data_type: DataType, callback: Callable) -> None
|
||||
|
||||
# CollectorManager API
|
||||
class CollectorManager:
|
||||
def add_collector(collector: BaseDataCollector) -> None
|
||||
async def start() -> bool
|
||||
async def stop() -> None
|
||||
def get_status() -> Dict[str, Any]
|
||||
def list_collectors() -> List[str]
|
||||
```
|
||||
|
||||
#### Exchange Factory APIs
|
||||
|
||||
```python
|
||||
# Factory Pattern API
|
||||
class ExchangeFactory:
|
||||
@staticmethod
|
||||
def create_collector(config: ExchangeCollectorConfig) -> BaseDataCollector
|
||||
|
||||
@staticmethod
|
||||
def create_multiple_collectors(configs: List[ExchangeCollectorConfig]) -> List[BaseDataCollector]
|
||||
|
||||
@staticmethod
|
||||
def get_supported_exchanges() -> List[str]
|
||||
|
||||
@staticmethod
|
||||
def validate_config(config: ExchangeCollectorConfig) -> bool
|
||||
|
||||
# Configuration API
|
||||
@dataclass
|
||||
class ExchangeCollectorConfig:
|
||||
exchange: str
|
||||
symbol: str
|
||||
data_types: List[DataType]
|
||||
auto_restart: bool = True
|
||||
health_check_interval: float = 30.0
|
||||
store_raw_data: bool = True
|
||||
custom_params: Optional[Dict[str, Any]] = None
|
||||
```
|
||||
|
||||
## 📊 Data Schemas
|
||||
|
||||
### Market Data Point
|
||||
|
||||
The standardized data structure for all market data:
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class MarketDataPoint:
|
||||
exchange: str # Exchange name (e.g., 'okx', 'binance')
|
||||
symbol: str # Trading symbol (e.g., 'BTC-USDT')
|
||||
timestamp: datetime # Data timestamp (UTC)
|
||||
data_type: DataType # Type of data (TRADE, ORDERBOOK, etc.)
|
||||
data: Dict[str, Any] # Raw data payload
|
||||
```
|
||||
|
||||
### Data Types
|
||||
|
||||
```python
|
||||
class DataType(Enum):
|
||||
TICKER = "ticker" # Price and volume updates
|
||||
TRADE = "trade" # Individual trade executions
|
||||
ORDERBOOK = "orderbook" # Order book snapshots
|
||||
CANDLE = "candle" # OHLCV candle data
|
||||
BALANCE = "balance" # Account balance updates
|
||||
```
|
||||
|
||||
### Status Schemas
|
||||
|
||||
#### Collector Status
|
||||
|
||||
```python
|
||||
{
|
||||
'exchange': str, # Exchange name
|
||||
'status': str, # Current status (running, stopped, error)
|
||||
'should_be_running': bool, # Desired state
|
||||
'symbols': List[str], # Configured symbols
|
||||
'data_types': List[str], # Data types being collected
|
||||
'auto_restart': bool, # Auto-restart enabled
|
||||
'health': {
|
||||
'time_since_heartbeat': float, # Seconds since last heartbeat
|
||||
'time_since_data': float, # Seconds since last data
|
||||
'max_silence_duration': float # Max allowed silence
|
||||
},
|
||||
'statistics': {
|
||||
'messages_received': int, # Total messages received
|
||||
'messages_processed': int, # Successfully processed
|
||||
'errors': int, # Error count
|
||||
'restarts': int, # Restart count
|
||||
'uptime_seconds': float, # Current uptime
|
||||
'reconnect_attempts': int, # Current reconnect attempts
|
||||
'last_message_time': str, # ISO timestamp
|
||||
'connection_uptime': str, # Connection start time
|
||||
'last_error': str, # Last error message
|
||||
'last_restart_time': str # Last restart time
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Health Status
|
||||
|
||||
```python
|
||||
{
|
||||
'is_healthy': bool, # Overall health status
|
||||
'issues': List[str], # List of current issues
|
||||
'status': str, # Current collector status
|
||||
'last_heartbeat': str, # Last heartbeat timestamp
|
||||
'last_data_received': str, # Last data timestamp
|
||||
'should_be_running': bool, # Expected state
|
||||
'is_running': bool # Actual running state
|
||||
}
|
||||
```
|
||||
|
||||
## 🔧 Configuration Schemas
|
||||
|
||||
### Database Configuration
|
||||
|
||||
```json
|
||||
{
|
||||
"database": {
|
||||
"url": "postgresql://user:pass@host:port/db",
|
||||
"pool_size": 10,
|
||||
"max_overflow": 20,
|
||||
"pool_timeout": 30,
|
||||
"pool_recycle": 3600
|
||||
},
|
||||
"tables": {
|
||||
"market_data": "market_data",
|
||||
"raw_trades": "raw_trades",
|
||||
"collector_status": "collector_status"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Exchange Configuration
|
||||
|
||||
```json
|
||||
{
|
||||
"exchange": "okx",
|
||||
"connection": {
|
||||
"public_ws_url": "wss://ws.okx.com:8443/ws/v5/public",
|
||||
"ping_interval": 25.0,
|
||||
"pong_timeout": 10.0,
|
||||
"max_reconnect_attempts": 5,
|
||||
"reconnect_delay": 5.0
|
||||
},
|
||||
"data_collection": {
|
||||
"store_raw_data": true,
|
||||
"health_check_interval": 30.0,
|
||||
"auto_restart": true,
|
||||
"buffer_size": 1000
|
||||
},
|
||||
"trading_pairs": [
|
||||
{
|
||||
"symbol": "BTC-USDT",
|
||||
"enabled": true,
|
||||
"data_types": ["trade", "orderbook"],
|
||||
"channels": {
|
||||
"trades": "trades",
|
||||
"orderbook": "books5",
|
||||
"ticker": "tickers"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Logging Configuration
|
||||
|
||||
```json
|
||||
{
|
||||
"logging": {
|
||||
"level": "INFO",
|
||||
"format": "detailed",
|
||||
"console_output": true,
|
||||
"file_output": true,
|
||||
"cleanup": true,
|
||||
"max_files": 30,
|
||||
"log_directory": "./logs"
|
||||
},
|
||||
"components": {
|
||||
"data_collectors": {
|
||||
"level": "INFO",
|
||||
"verbose": false
|
||||
},
|
||||
"websocket_clients": {
|
||||
"level": "DEBUG",
|
||||
"verbose": true
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🌐 Protocol Specifications
|
||||
|
||||
### WebSocket Message Formats
|
||||
|
||||
#### OKX Message Format
|
||||
|
||||
```json
|
||||
{
|
||||
"arg": {
|
||||
"channel": "trades",
|
||||
"instId": "BTC-USDT"
|
||||
},
|
||||
"data": [
|
||||
{
|
||||
"instId": "BTC-USDT",
|
||||
"tradeId": "12345678",
|
||||
"px": "50000.5",
|
||||
"sz": "0.001",
|
||||
"side": "buy",
|
||||
"ts": "1697123456789"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### Subscription Message Format
|
||||
|
||||
```json
|
||||
{
|
||||
"op": "subscribe",
|
||||
"args": [
|
||||
{
|
||||
"channel": "trades",
|
||||
"instId": "BTC-USDT"
|
||||
},
|
||||
{
|
||||
"channel": "books5",
|
||||
"instId": "BTC-USDT"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Database Schemas
|
||||
|
||||
#### Market Data Table
|
||||
|
||||
```sql
|
||||
CREATE TABLE market_data (
|
||||
id SERIAL PRIMARY KEY,
|
||||
exchange VARCHAR(50) NOT NULL,
|
||||
symbol VARCHAR(50) NOT NULL,
|
||||
data_type VARCHAR(20) NOT NULL,
|
||||
timestamp TIMESTAMP WITH TIME ZONE NOT NULL,
|
||||
data JSONB NOT NULL,
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
|
||||
INDEX(exchange, symbol, timestamp),
|
||||
INDEX(data_type, timestamp)
|
||||
);
|
||||
```
|
||||
|
||||
#### Raw Trades Table
|
||||
|
||||
```sql
|
||||
CREATE TABLE raw_trades (
|
||||
id SERIAL PRIMARY KEY,
|
||||
exchange VARCHAR(50) NOT NULL,
|
||||
symbol VARCHAR(50) NOT NULL,
|
||||
trade_id VARCHAR(100),
|
||||
price DECIMAL(20, 8) NOT NULL,
|
||||
size DECIMAL(20, 8) NOT NULL,
|
||||
side VARCHAR(10) NOT NULL,
|
||||
timestamp TIMESTAMP WITH TIME ZONE NOT NULL,
|
||||
raw_data JSONB,
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
|
||||
UNIQUE(exchange, symbol, trade_id),
|
||||
INDEX(exchange, symbol, timestamp),
|
||||
INDEX(timestamp)
|
||||
);
|
||||
```
|
||||
|
||||
## 📈 Performance Specifications
|
||||
|
||||
### System Requirements
|
||||
|
||||
#### Minimum Requirements
|
||||
|
||||
- **CPU**: 2 cores, 2.0 GHz
|
||||
- **Memory**: 4 GB RAM
|
||||
- **Storage**: 20 GB available space
|
||||
- **Network**: Stable internet connection (100 Mbps+)
|
||||
|
||||
#### Recommended Requirements
|
||||
|
||||
- **CPU**: 4+ cores, 3.0+ GHz
|
||||
- **Memory**: 8+ GB RAM
|
||||
- **Storage**: 100+ GB SSD
|
||||
- **Network**: High-speed internet (1 Gbps+)
|
||||
|
||||
### Performance Targets
|
||||
|
||||
#### Data Collection
|
||||
|
||||
- **Latency**: < 100ms from exchange to processing
|
||||
- **Throughput**: 1000+ messages/second per collector
|
||||
- **Uptime**: 99.9% availability
|
||||
- **Memory Usage**: < 50 MB per collector
|
||||
|
||||
#### Database Operations
|
||||
|
||||
- **Insert Rate**: 10,000+ inserts/second
|
||||
- **Query Response**: < 100ms for typical queries
|
||||
- **Storage Growth**: ~1 GB/month per active trading pair
|
||||
- **Retention**: 2+ years of historical data
|
||||
|
||||
## 🔒 Security Specifications
|
||||
|
||||
### Authentication & Authorization
|
||||
|
||||
- **API Keys**: Secure storage in environment variables
|
||||
- **Database Access**: Connection pooling with authentication
|
||||
- **WebSocket Connections**: TLS encryption for all connections
|
||||
- **Logging**: No sensitive data in logs
|
||||
|
||||
### Data Protection
|
||||
|
||||
- **Encryption**: TLS 1.3 for all external communications
|
||||
- **Data Validation**: Comprehensive input validation
|
||||
- **Error Handling**: Secure error messages without data leakage
|
||||
- **Backup**: Regular automated backups with encryption
|
||||
|
||||
## 🔗 Related Documentation
|
||||
|
||||
- **[Components Documentation](../components/)** - Implementation details
|
||||
- **[Architecture Overview](../architecture/)** - System design
|
||||
- **[Exchange Documentation](../exchanges/)** - Exchange integrations
|
||||
- **[Setup Guide](../guides/)** - Configuration and deployment
|
||||
|
||||
## 📞 Support
|
||||
|
||||
### API Support
|
||||
|
||||
For API-related questions:
|
||||
|
||||
1. **Check Examples**: Review code examples in each API section
|
||||
2. **Test Endpoints**: Use provided test scripts
|
||||
3. **Validate Schemas**: Ensure data matches specified formats
|
||||
4. **Review Logs**: Check detailed logs for API interactions
|
||||
|
||||
### Schema Validation
|
||||
|
||||
For data schema issues:
|
||||
|
||||
```python
|
||||
# Validate data point structure
|
||||
def validate_market_data_point(data_point):
|
||||
required_fields = ['exchange', 'symbol', 'timestamp', 'data_type', 'data']
|
||||
for field in required_fields:
|
||||
if not hasattr(data_point, field):
|
||||
raise ValueError(f"Missing required field: {field}")
|
||||
|
||||
if not isinstance(data_point.data_type, DataType):
|
||||
raise ValueError("Invalid data_type")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
*For the complete documentation index, see the [main documentation README](../README.md).*
|
||||
Loading…
x
Reference in New Issue
Block a user