documentation
This commit is contained in:
Vasily.onl
297
docs/exchanges/README.md
Normal file
297
docs/exchanges/README.md
Normal file
@@ -0,0 +1,297 @@
|
||||
# Exchange Documentation
|
||||
|
||||
This section contains detailed documentation for all cryptocurrency exchange integrations in the TCP Dashboard platform.
|
||||
|
||||
## 📋 Contents
|
||||
|
||||
### Supported Exchanges
|
||||
|
||||
#### Production Ready
|
||||
|
||||
- **[OKX Collector](okx_collector.md)** - *Complete guide to OKX exchange integration*
|
||||
- Real-time trades, orderbook, and ticker data collection
|
||||
- WebSocket connection management with OKX-specific ping/pong
|
||||
- Factory pattern usage and configuration
|
||||
- Data processing and validation
|
||||
- Monitoring and troubleshooting
|
||||
- Production deployment guide
|
||||
|
||||
#### Planned Integrations
|
||||
|
||||
- **Binance** - Major global exchange (development planned)
|
||||
- **Coinbase Pro** - US-regulated exchange (development planned)
|
||||
- **Kraken** - European exchange (development planned)
|
||||
- **Bybit** - Derivatives exchange (development planned)
|
||||
|
||||
## 🏗️ Exchange Architecture
|
||||
|
||||
### Modular Design
|
||||
|
||||
Each exchange implementation follows a standardized structure:
|
||||
|
||||
```
|
||||
data/exchanges/
|
||||
├── __init__.py # Main exports and factory
|
||||
├── registry.py # Exchange registry and capabilities
|
||||
├── factory.py # Factory pattern for collectors
|
||||
└── {exchange}/ # Exchange-specific implementation
|
||||
├── __init__.py # Exchange exports
|
||||
├── collector.py # {Exchange}Collector class
|
||||
└── websocket.py # {Exchange}WebSocketClient class
|
||||
```
|
||||
|
||||
### Standardized Interface
|
||||
|
||||
All exchange collectors implement the same interface:
|
||||
|
||||
```python
|
||||
from data.exchanges import ExchangeFactory, ExchangeCollectorConfig
|
||||
from data.base_collector import DataType
|
||||
|
||||
# Unified configuration across all exchanges
|
||||
config = ExchangeCollectorConfig(
|
||||
exchange='okx', # or 'binance', 'coinbase', etc.
|
||||
symbol='BTC-USDT',
|
||||
data_types=[DataType.TRADE, DataType.ORDERBOOK],
|
||||
auto_restart=True
|
||||
)
|
||||
|
||||
collector = ExchangeFactory.create_collector(config)
|
||||
```
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### Using Factory Pattern
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from data.exchanges import get_supported_exchanges, create_okx_collector
|
||||
from data.base_collector import DataType
|
||||
|
||||
async def main():
|
||||
# Check supported exchanges
|
||||
exchanges = get_supported_exchanges()
|
||||
print(f"Supported: {exchanges}") # ['okx']
|
||||
|
||||
# Create OKX collector
|
||||
collector = create_okx_collector(
|
||||
symbol='BTC-USDT',
|
||||
data_types=[DataType.TRADE, DataType.ORDERBOOK]
|
||||
)
|
||||
|
||||
# Add data callback
|
||||
def on_trade(data_point):
|
||||
print(f"Trade: {data_point.data}")
|
||||
|
||||
collector.add_data_callback(DataType.TRADE, on_trade)
|
||||
|
||||
# Start collection
|
||||
await collector.start()
|
||||
await asyncio.sleep(60)
|
||||
await collector.stop()
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Multi-Exchange Setup
|
||||
|
||||
```python
|
||||
from data.exchanges import ExchangeFactory, ExchangeCollectorConfig
|
||||
from data.collector_manager import CollectorManager
|
||||
|
||||
async def setup_multi_exchange():
|
||||
manager = CollectorManager("multi_exchange_system")
|
||||
|
||||
# Future: Multiple exchanges
|
||||
configs = [
|
||||
ExchangeCollectorConfig('okx', 'BTC-USDT', [DataType.TRADE]),
|
||||
# ExchangeCollectorConfig('binance', 'BTC-USDT', [DataType.TRADE]),
|
||||
# ExchangeCollectorConfig('coinbase', 'BTC-USD', [DataType.TRADE])
|
||||
]
|
||||
|
||||
for config in configs:
|
||||
collector = ExchangeFactory.create_collector(config)
|
||||
manager.add_collector(collector)
|
||||
|
||||
await manager.start()
|
||||
return manager
|
||||
```
|
||||
|
||||
## 📊 Exchange Capabilities
|
||||
|
||||
### Data Types
|
||||
|
||||
Different exchanges support different data types:
|
||||
|
||||
| Exchange | Trades | Orderbook | Ticker | Candles | Balance |
|
||||
|----------|--------|-----------|--------|---------|---------|
|
||||
| OKX | ✅ | ✅ | ✅ | 🔄 | 🔄 |
|
||||
| Binance | 🔄 | 🔄 | 🔄 | 🔄 | 🔄 |
|
||||
| Coinbase | 🔄 | 🔄 | 🔄 | 🔄 | 🔄 |
|
||||
|
||||
Legend: ✅ Implemented, 🔄 Planned, ❌ Not supported
|
||||
|
||||
### Trading Pairs
|
||||
|
||||
Query supported trading pairs for each exchange:
|
||||
|
||||
```python
|
||||
from data.exchanges import ExchangeFactory
|
||||
|
||||
# Get supported pairs
|
||||
okx_pairs = ExchangeFactory.get_supported_pairs('okx')
|
||||
print(f"OKX pairs: {okx_pairs}")
|
||||
|
||||
# Get exchange information
|
||||
okx_info = ExchangeFactory.get_exchange_info('okx')
|
||||
print(f"OKX capabilities: {okx_info}")
|
||||
```
|
||||
|
||||
## 🔧 Exchange Configuration
|
||||
|
||||
### Common Configuration
|
||||
|
||||
All exchanges share common configuration options:
|
||||
|
||||
```python
|
||||
from data.exchanges import ExchangeCollectorConfig
|
||||
|
||||
config = ExchangeCollectorConfig(
|
||||
exchange='okx', # Exchange name
|
||||
symbol='BTC-USDT', # Trading pair
|
||||
data_types=[DataType.TRADE], # Data types to collect
|
||||
auto_restart=True, # Auto-restart on failures
|
||||
health_check_interval=30.0, # Health check interval
|
||||
store_raw_data=True, # Store raw exchange data
|
||||
custom_params={ # Exchange-specific parameters
|
||||
'ping_interval': 25.0,
|
||||
'max_reconnect_attempts': 5
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
### Exchange-Specific Configuration
|
||||
|
||||
Each exchange has specific configuration files:
|
||||
|
||||
- **OKX**: `config/okx_config.json`
|
||||
- **Binance**: `config/binance_config.json` (planned)
|
||||
- **Coinbase**: `config/coinbase_config.json` (planned)
|
||||
|
||||
## 📈 Performance Comparison
|
||||
|
||||
### Real-time Data Rates
|
||||
|
||||
Approximate message rates for different exchanges:
|
||||
|
||||
| Exchange | Trades/sec | Orderbook Updates/sec | Latency |
|
||||
|----------|------------|----------------------|---------|
|
||||
| OKX | 5-50 | 10-100 | ~50ms |
|
||||
| Binance | TBD | TBD | TBD |
|
||||
| Coinbase | TBD | TBD | TBD |
|
||||
|
||||
*Note: Rates vary by trading pair activity*
|
||||
|
||||
### Resource Usage
|
||||
|
||||
Memory and CPU usage per collector:
|
||||
|
||||
| Exchange | Memory (MB) | CPU (%) | Network (KB/s) |
|
||||
|----------|-------------|---------|----------------|
|
||||
| OKX | 15-25 | 1-3 | 5-20 |
|
||||
| Binance | TBD | TBD | TBD |
|
||||
| Coinbase | TBD | TBD | TBD |
|
||||
|
||||
## 🔍 Monitoring & Debugging
|
||||
|
||||
### Exchange Status
|
||||
|
||||
Monitor exchange-specific metrics:
|
||||
|
||||
```python
|
||||
# Get exchange status
|
||||
status = collector.get_status()
|
||||
print(f"Exchange: {status['exchange']}")
|
||||
print(f"WebSocket State: {status['websocket_state']}")
|
||||
print(f"Messages Processed: {status['messages_processed']}")
|
||||
|
||||
# Exchange-specific metrics
|
||||
if 'websocket_stats' in status:
|
||||
ws_stats = status['websocket_stats']
|
||||
print(f"Reconnections: {ws_stats['reconnections']}")
|
||||
print(f"Ping/Pong: {ws_stats['pings_sent']}/{ws_stats['pongs_received']}")
|
||||
```
|
||||
|
||||
### Debug Mode
|
||||
|
||||
Enable exchange-specific debugging:
|
||||
|
||||
```python
|
||||
import os
|
||||
os.environ['LOG_LEVEL'] = 'DEBUG'
|
||||
|
||||
# Detailed exchange logging
|
||||
collector = create_okx_collector('BTC-USDT', [DataType.TRADE])
|
||||
# Check logs: ./logs/okx_collector_btc_usdt_debug.log
|
||||
```
|
||||
|
||||
## 🛠️ Adding New Exchanges
|
||||
|
||||
### Implementation Checklist
|
||||
|
||||
To add a new exchange:
|
||||
|
||||
1. **Create Exchange Folder**: `data/exchanges/{exchange}/`
|
||||
2. **Implement WebSocket Client**: `{exchange}/websocket.py`
|
||||
3. **Implement Collector**: `{exchange}/collector.py`
|
||||
4. **Add to Registry**: Update `registry.py`
|
||||
5. **Create Configuration**: `config/{exchange}_config.json`
|
||||
6. **Add Documentation**: `docs/exchanges/{exchange}_collector.md`
|
||||
7. **Add Tests**: `tests/test_{exchange}_collector.py`
|
||||
|
||||
### Implementation Template
|
||||
|
||||
```python
|
||||
# data/exchanges/newexchange/collector.py
|
||||
from data.base_collector import BaseDataCollector, DataType
|
||||
from .websocket import NewExchangeWebSocketClient
|
||||
|
||||
class NewExchangeCollector(BaseDataCollector):
|
||||
def __init__(self, symbol: str, **kwargs):
|
||||
super().__init__("newexchange", [symbol], **kwargs)
|
||||
self.ws_client = NewExchangeWebSocketClient()
|
||||
|
||||
async def connect(self) -> bool:
|
||||
return await self.ws_client.connect()
|
||||
|
||||
# Implement other required methods...
|
||||
```
|
||||
|
||||
## 🔗 Related Documentation
|
||||
|
||||
- **[Components Documentation](../components/)** - Core system components
|
||||
- **[Architecture Overview](../architecture/)** - System design
|
||||
- **[Setup Guide](../guides/setup.md)** - Configuration and deployment
|
||||
- **[API Reference](../reference/)** - Technical specifications
|
||||
|
||||
## 📞 Support
|
||||
|
||||
### Exchange-Specific Issues
|
||||
|
||||
For exchange-specific problems:
|
||||
|
||||
1. **Check Status**: Use `get_status()` and `get_health_status()`
|
||||
2. **Review Logs**: Check exchange-specific log files
|
||||
3. **Verify Configuration**: Confirm exchange configuration files
|
||||
4. **Test Connection**: Run exchange-specific test scripts
|
||||
|
||||
### Common Issues
|
||||
|
||||
- **Rate Limiting**: Each exchange has different rate limits
|
||||
- **Symbol Formats**: Trading pair naming conventions vary
|
||||
- **WebSocket Protocols**: Each exchange has unique WebSocket requirements
|
||||
- **Data Formats**: Message structures differ between exchanges
|
||||
|
||||
---
|
||||
|
||||
*For the complete documentation index, see the [main documentation README](../README.md).*
|
||||
945
docs/exchanges/okx_collector.md
Normal file
945
docs/exchanges/okx_collector.md
Normal file
@@ -0,0 +1,945 @@
|
||||
# OKX Data Collector Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
The OKX Data Collector provides real-time market data collection from OKX exchange using WebSocket API. It's built on the modular exchange architecture and provides robust connection management, automatic reconnection, health monitoring, and comprehensive data processing.
|
||||
|
||||
## Features
|
||||
|
||||
### 🎯 **OKX-Specific Features**
|
||||
- **Real-time Data**: Live trades, orderbook, and ticker data
|
||||
- **Single Pair Focus**: Each collector handles one trading pair for better isolation
|
||||
- **Ping/Pong Management**: OKX-specific keepalive mechanism with proper format
|
||||
- **Raw Data Storage**: Optional storage of raw OKX messages for debugging
|
||||
- **Connection Resilience**: Robust reconnection logic for OKX WebSocket
|
||||
|
||||
### 📊 **Supported Data Types**
|
||||
- **Trades**: Real-time trade executions (`trades` channel)
|
||||
- **Orderbook**: 5-level order book depth (`books5` channel)
|
||||
- **Ticker**: 24h ticker statistics (`tickers` channel)
|
||||
- **Future**: Candle data support planned
|
||||
|
||||
### 🔧 **Configuration Options**
|
||||
- Auto-restart on failures
|
||||
- Health check intervals
|
||||
- Raw data storage toggle
|
||||
- Custom ping/pong timing
|
||||
- Reconnection attempts configuration
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Using Factory Pattern (Recommended)
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from data.exchanges import create_okx_collector
|
||||
from data.base_collector import DataType
|
||||
|
||||
async def main():
|
||||
# Create OKX collector using convenience function
|
||||
collector = create_okx_collector(
|
||||
symbol='BTC-USDT',
|
||||
data_types=[DataType.TRADE, DataType.ORDERBOOK],
|
||||
auto_restart=True,
|
||||
health_check_interval=30.0,
|
||||
store_raw_data=True
|
||||
)
|
||||
|
||||
# Add data callbacks
|
||||
def on_trade(data_point):
|
||||
trade = data_point.data
|
||||
print(f"Trade: {trade['side']} {trade['sz']} @ {trade['px']} (ID: {trade['tradeId']})")
|
||||
|
||||
def on_orderbook(data_point):
|
||||
book = data_point.data
|
||||
if book.get('bids') and book.get('asks'):
|
||||
best_bid = book['bids'][0]
|
||||
best_ask = book['asks'][0]
|
||||
print(f"Orderbook: Bid {best_bid[0]}@{best_bid[1]} Ask {best_ask[0]}@{best_ask[1]}")
|
||||
|
||||
collector.add_data_callback(DataType.TRADE, on_trade)
|
||||
collector.add_data_callback(DataType.ORDERBOOK, on_orderbook)
|
||||
|
||||
# Start collector
|
||||
await collector.start()
|
||||
|
||||
# Run for 60 seconds
|
||||
await asyncio.sleep(60)
|
||||
|
||||
# Stop gracefully
|
||||
await collector.stop()
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### 2. Direct OKX Collector Usage
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from data.exchanges.okx import OKXCollector
|
||||
from data.base_collector import DataType
|
||||
|
||||
async def main():
|
||||
# Create collector directly
|
||||
collector = OKXCollector(
|
||||
symbol='ETH-USDT',
|
||||
data_types=[DataType.TRADE, DataType.ORDERBOOK],
|
||||
component_name='eth_collector',
|
||||
auto_restart=True,
|
||||
health_check_interval=30.0,
|
||||
store_raw_data=True
|
||||
)
|
||||
|
||||
# Add callbacks
|
||||
def on_data(data_point):
|
||||
print(f"{data_point.data_type.value}: {data_point.symbol} - {data_point.timestamp}")
|
||||
|
||||
collector.add_data_callback(DataType.TRADE, on_data)
|
||||
collector.add_data_callback(DataType.ORDERBOOK, on_data)
|
||||
|
||||
# Start and monitor
|
||||
await collector.start()
|
||||
|
||||
# Monitor status
|
||||
for i in range(12): # 60 seconds total
|
||||
await asyncio.sleep(5)
|
||||
status = collector.get_status()
|
||||
print(f"Status: {status['status']} - Messages: {status.get('messages_processed', 0)}")
|
||||
|
||||
await collector.stop()
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### 3. Multiple OKX Collectors with Manager
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from data.collector_manager import CollectorManager
|
||||
from data.exchanges import create_okx_collector
|
||||
from data.base_collector import DataType
|
||||
|
||||
async def main():
|
||||
# Create manager
|
||||
manager = CollectorManager(
|
||||
manager_name="okx_trading_system",
|
||||
global_health_check_interval=30.0
|
||||
)
|
||||
|
||||
# Create multiple OKX collectors
|
||||
symbols = ['BTC-USDT', 'ETH-USDT', 'SOL-USDT']
|
||||
|
||||
for symbol in symbols:
|
||||
collector = create_okx_collector(
|
||||
symbol=symbol,
|
||||
data_types=[DataType.TRADE, DataType.ORDERBOOK],
|
||||
auto_restart=True
|
||||
)
|
||||
manager.add_collector(collector)
|
||||
|
||||
# Start manager
|
||||
await manager.start()
|
||||
|
||||
# Monitor all collectors
|
||||
while True:
|
||||
status = manager.get_status()
|
||||
stats = status.get('statistics', {})
|
||||
|
||||
print(f"=== OKX Collectors Status ===")
|
||||
print(f"Running: {stats.get('running_collectors', 0)}")
|
||||
print(f"Failed: {stats.get('failed_collectors', 0)}")
|
||||
print(f"Total messages: {stats.get('total_messages', 0)}")
|
||||
|
||||
# Individual collector status
|
||||
for collector_name in manager.list_collectors():
|
||||
collector_status = manager.get_collector_status(collector_name)
|
||||
if collector_status:
|
||||
info = collector_status.get('status', {})
|
||||
print(f" {collector_name}: {info.get('status')} - "
|
||||
f"Messages: {info.get('messages_processed', 0)}")
|
||||
|
||||
await asyncio.sleep(15)
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### 1. JSON Configuration File
|
||||
|
||||
The system uses `config/okx_config.json` for configuration:
|
||||
|
||||
```json
|
||||
{
|
||||
"exchange": "okx",
|
||||
"connection": {
|
||||
"public_ws_url": "wss://ws.okx.com:8443/ws/v5/public",
|
||||
"private_ws_url": "wss://ws.okx.com:8443/ws/v5/private",
|
||||
"ping_interval": 25.0,
|
||||
"pong_timeout": 10.0,
|
||||
"max_reconnect_attempts": 5,
|
||||
"reconnect_delay": 5.0
|
||||
},
|
||||
"data_collection": {
|
||||
"store_raw_data": true,
|
||||
"health_check_interval": 30.0,
|
||||
"auto_restart": true,
|
||||
"buffer_size": 1000
|
||||
},
|
||||
"factory": {
|
||||
"use_factory_pattern": true,
|
||||
"default_data_types": ["trade", "orderbook"],
|
||||
"batch_create": true
|
||||
},
|
||||
"trading_pairs": [
|
||||
{
|
||||
"symbol": "BTC-USDT",
|
||||
"enabled": true,
|
||||
"data_types": ["trade", "orderbook"],
|
||||
"channels": {
|
||||
"trades": "trades",
|
||||
"orderbook": "books5",
|
||||
"ticker": "tickers"
|
||||
}
|
||||
},
|
||||
{
|
||||
"symbol": "ETH-USDT",
|
||||
"enabled": true,
|
||||
"data_types": ["trade", "orderbook"],
|
||||
"channels": {
|
||||
"trades": "trades",
|
||||
"orderbook": "books5",
|
||||
"ticker": "tickers"
|
||||
}
|
||||
}
|
||||
],
|
||||
"logging": {
|
||||
"component_name_template": "okx_collector_{symbol}",
|
||||
"log_level": "INFO",
|
||||
"verbose": false
|
||||
},
|
||||
"database": {
|
||||
"store_processed_data": true,
|
||||
"store_raw_data": true,
|
||||
"batch_size": 100,
|
||||
"flush_interval": 5.0
|
||||
},
|
||||
"monitoring": {
|
||||
"enable_health_checks": true,
|
||||
"health_check_interval": 30.0,
|
||||
"alert_on_connection_loss": true,
|
||||
"max_consecutive_errors": 5
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Programmatic Configuration
|
||||
|
||||
```python
|
||||
from data.exchanges.okx import OKXCollector
|
||||
from data.base_collector import DataType
|
||||
|
||||
# Custom configuration
|
||||
collector = OKXCollector(
|
||||
symbol='BTC-USDT',
|
||||
data_types=[DataType.TRADE, DataType.ORDERBOOK],
|
||||
component_name='custom_btc_collector',
|
||||
auto_restart=True,
|
||||
health_check_interval=15.0, # Check every 15 seconds
|
||||
store_raw_data=True # Store raw OKX messages
|
||||
)
|
||||
```
|
||||
|
||||
### 3. Factory Configuration
|
||||
|
||||
```python
|
||||
from data.exchanges import ExchangeFactory, ExchangeCollectorConfig
|
||||
from data.base_collector import DataType
|
||||
|
||||
config = ExchangeCollectorConfig(
|
||||
exchange='okx',
|
||||
symbol='ETH-USDT',
|
||||
data_types=[DataType.TRADE, DataType.ORDERBOOK],
|
||||
auto_restart=True,
|
||||
health_check_interval=30.0,
|
||||
store_raw_data=True,
|
||||
custom_params={
|
||||
'ping_interval': 20.0, # Custom ping interval
|
||||
'max_reconnect_attempts': 10, # More reconnection attempts
|
||||
'pong_timeout': 15.0 # Longer pong timeout
|
||||
}
|
||||
)
|
||||
|
||||
collector = ExchangeFactory.create_collector(config)
|
||||
```
|
||||
|
||||
## Data Processing
|
||||
|
||||
### OKX Message Formats
|
||||
|
||||
#### Trade Data
|
||||
|
||||
```python
|
||||
# Raw OKX trade message
|
||||
{
|
||||
"arg": {
|
||||
"channel": "trades",
|
||||
"instId": "BTC-USDT"
|
||||
},
|
||||
"data": [
|
||||
{
|
||||
"instId": "BTC-USDT",
|
||||
"tradeId": "12345678",
|
||||
"px": "50000.5", # Price
|
||||
"sz": "0.001", # Size
|
||||
"side": "buy", # Side (buy/sell)
|
||||
"ts": "1697123456789" # Timestamp (ms)
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
# Processed MarketDataPoint
|
||||
MarketDataPoint(
|
||||
exchange="okx",
|
||||
symbol="BTC-USDT",
|
||||
timestamp=datetime(2023, 10, 12, 15, 30, 56, tzinfo=timezone.utc),
|
||||
data_type=DataType.TRADE,
|
||||
data={
|
||||
"instId": "BTC-USDT",
|
||||
"tradeId": "12345678",
|
||||
"px": "50000.5",
|
||||
"sz": "0.001",
|
||||
"side": "buy",
|
||||
"ts": "1697123456789"
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
#### Orderbook Data
|
||||
|
||||
```python
|
||||
# Raw OKX orderbook message (books5)
|
||||
{
|
||||
"arg": {
|
||||
"channel": "books5",
|
||||
"instId": "BTC-USDT"
|
||||
},
|
||||
"data": [
|
||||
{
|
||||
"asks": [
|
||||
["50001.0", "0.5", "0", "3"], # [price, size, liquidated, orders]
|
||||
["50002.0", "1.0", "0", "5"]
|
||||
],
|
||||
"bids": [
|
||||
["50000.0", "0.8", "0", "2"],
|
||||
["49999.0", "1.2", "0", "4"]
|
||||
],
|
||||
"ts": "1697123456789",
|
||||
"checksum": "123456789"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
# Usage in callback
|
||||
def on_orderbook(data_point):
|
||||
book = data_point.data
|
||||
|
||||
if book.get('bids') and book.get('asks'):
|
||||
best_bid = book['bids'][0]
|
||||
best_ask = book['asks'][0]
|
||||
|
||||
spread = float(best_ask[0]) - float(best_bid[0])
|
||||
print(f"Spread: ${spread:.2f}")
|
||||
```
|
||||
|
||||
#### Ticker Data
|
||||
|
||||
```python
|
||||
# Raw OKX ticker message
|
||||
{
|
||||
"arg": {
|
||||
"channel": "tickers",
|
||||
"instId": "BTC-USDT"
|
||||
},
|
||||
"data": [
|
||||
{
|
||||
"instType": "SPOT",
|
||||
"instId": "BTC-USDT",
|
||||
"last": "50000.5", # Last price
|
||||
"lastSz": "0.001", # Last size
|
||||
"askPx": "50001.0", # Best ask price
|
||||
"askSz": "0.5", # Best ask size
|
||||
"bidPx": "50000.0", # Best bid price
|
||||
"bidSz": "0.8", # Best bid size
|
||||
"open24h": "49500.0", # 24h open
|
||||
"high24h": "50500.0", # 24h high
|
||||
"low24h": "49000.0", # 24h low
|
||||
"vol24h": "1234.567", # 24h volume
|
||||
"ts": "1697123456789"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Data Validation
|
||||
|
||||
The OKX collector includes comprehensive data validation:
|
||||
|
||||
```python
|
||||
# Automatic validation in collector
|
||||
class OKXCollector(BaseDataCollector):
|
||||
async def _process_data_item(self, channel: str, data_item: Dict[str, Any]):
|
||||
# Validate message structure
|
||||
if not isinstance(data_item, dict):
|
||||
self.logger.warning("Invalid data item type")
|
||||
return None
|
||||
|
||||
# Validate required fields based on channel
|
||||
if channel == "trades":
|
||||
required_fields = ['tradeId', 'px', 'sz', 'side', 'ts']
|
||||
elif channel == "books5":
|
||||
required_fields = ['bids', 'asks', 'ts']
|
||||
elif channel == "tickers":
|
||||
required_fields = ['last', 'ts']
|
||||
else:
|
||||
self.logger.warning(f"Unknown channel: {channel}")
|
||||
return None
|
||||
|
||||
# Check required fields
|
||||
for field in required_fields:
|
||||
if field not in data_item:
|
||||
self.logger.warning(f"Missing required field '{field}' in {channel} data")
|
||||
return None
|
||||
|
||||
# Process and return validated data
|
||||
return await self._create_market_data_point(channel, data_item)
|
||||
```
|
||||
|
||||
## Monitoring and Status
|
||||
|
||||
### Status Information
|
||||
|
||||
```python
|
||||
# Get comprehensive status
|
||||
status = collector.get_status()
|
||||
|
||||
print(f"Exchange: {status['exchange']}") # 'okx'
|
||||
print(f"Symbol: {status['symbol']}") # 'BTC-USDT'
|
||||
print(f"Status: {status['status']}") # 'running'
|
||||
print(f"WebSocket Connected: {status['websocket_connected']}") # True/False
|
||||
print(f"WebSocket State: {status['websocket_state']}") # 'connected'
|
||||
print(f"Messages Processed: {status['messages_processed']}") # Integer
|
||||
print(f"Errors: {status['errors']}") # Integer
|
||||
print(f"Last Trade ID: {status['last_trade_id']}") # String or None
|
||||
|
||||
# WebSocket statistics
|
||||
if 'websocket_stats' in status:
|
||||
ws_stats = status['websocket_stats']
|
||||
print(f"Messages Received: {ws_stats['messages_received']}")
|
||||
print(f"Messages Sent: {ws_stats['messages_sent']}")
|
||||
print(f"Pings Sent: {ws_stats['pings_sent']}")
|
||||
print(f"Pongs Received: {ws_stats['pongs_received']}")
|
||||
print(f"Reconnections: {ws_stats['reconnections']}")
|
||||
```
|
||||
|
||||
### Health Monitoring
|
||||
|
||||
```python
|
||||
# Get health status
|
||||
health = collector.get_health_status()
|
||||
|
||||
print(f"Is Healthy: {health['is_healthy']}") # True/False
|
||||
print(f"Issues: {health['issues']}") # List of issues
|
||||
print(f"Last Heartbeat: {health['last_heartbeat']}") # ISO timestamp
|
||||
print(f"Last Data: {health['last_data_received']}") # ISO timestamp
|
||||
print(f"Should Be Running: {health['should_be_running']}") # True/False
|
||||
print(f"Is Running: {health['is_running']}") # True/False
|
||||
|
||||
# Auto-restart status
|
||||
if not health['is_healthy']:
|
||||
print("Collector is unhealthy - auto-restart will trigger")
|
||||
for issue in health['issues']:
|
||||
print(f" Issue: {issue}")
|
||||
```
|
||||
|
||||
### Performance Monitoring
|
||||
|
||||
```python
|
||||
import time
|
||||
|
||||
async def monitor_performance():
|
||||
collector = create_okx_collector('BTC-USDT', [DataType.TRADE])
|
||||
await collector.start()
|
||||
|
||||
start_time = time.time()
|
||||
last_message_count = 0
|
||||
|
||||
while True:
|
||||
await asyncio.sleep(10) # Check every 10 seconds
|
||||
|
||||
status = collector.get_status()
|
||||
current_messages = status.get('messages_processed', 0)
|
||||
|
||||
# Calculate message rate
|
||||
elapsed = time.time() - start_time
|
||||
messages_per_second = current_messages / elapsed if elapsed > 0 else 0
|
||||
|
||||
# Calculate recent rate
|
||||
recent_messages = current_messages - last_message_count
|
||||
recent_rate = recent_messages / 10 # per second over last 10 seconds
|
||||
|
||||
print(f"=== Performance Stats ===")
|
||||
print(f"Total Messages: {current_messages}")
|
||||
print(f"Average Rate: {messages_per_second:.2f} msg/sec")
|
||||
print(f"Recent Rate: {recent_rate:.2f} msg/sec")
|
||||
print(f"Errors: {status.get('errors', 0)}")
|
||||
print(f"WebSocket State: {status.get('websocket_state', 'unknown')}")
|
||||
|
||||
last_message_count = current_messages
|
||||
|
||||
# Run performance monitoring
|
||||
asyncio.run(monitor_performance())
|
||||
```
|
||||
|
||||
## WebSocket Connection Details
|
||||
|
||||
### OKX WebSocket Client
|
||||
|
||||
The OKX implementation includes a specialized WebSocket client:
|
||||
|
||||
```python
|
||||
from data.exchanges.okx import OKXWebSocketClient, OKXSubscription, OKXChannelType
|
||||
|
||||
# Create WebSocket client directly (usually handled by collector)
|
||||
ws_client = OKXWebSocketClient(
|
||||
component_name='okx_ws_btc',
|
||||
ping_interval=25.0, # Must be < 30 seconds for OKX
|
||||
pong_timeout=10.0,
|
||||
max_reconnect_attempts=5,
|
||||
reconnect_delay=5.0
|
||||
)
|
||||
|
||||
# Connect to OKX
|
||||
await ws_client.connect(use_public=True)
|
||||
|
||||
# Create subscriptions
|
||||
subscriptions = [
|
||||
OKXSubscription(
|
||||
channel=OKXChannelType.TRADES.value,
|
||||
inst_id='BTC-USDT',
|
||||
enabled=True
|
||||
),
|
||||
OKXSubscription(
|
||||
channel=OKXChannelType.BOOKS5.value,
|
||||
inst_id='BTC-USDT',
|
||||
enabled=True
|
||||
)
|
||||
]
|
||||
|
||||
# Subscribe to channels
|
||||
await ws_client.subscribe(subscriptions)
|
||||
|
||||
# Add message callback
|
||||
def on_message(message):
|
||||
print(f"Received: {message}")
|
||||
|
||||
ws_client.add_message_callback(on_message)
|
||||
|
||||
# WebSocket will handle messages automatically
|
||||
await asyncio.sleep(60)
|
||||
|
||||
# Disconnect
|
||||
await ws_client.disconnect()
|
||||
```
|
||||
|
||||
### Connection States
|
||||
|
||||
The WebSocket client tracks connection states:
|
||||
|
||||
```python
|
||||
from data.exchanges.okx.websocket import ConnectionState
|
||||
|
||||
# Check connection state
|
||||
state = ws_client.connection_state
|
||||
|
||||
if state == ConnectionState.CONNECTED:
|
||||
print("WebSocket is connected and ready")
|
||||
elif state == ConnectionState.CONNECTING:
|
||||
print("WebSocket is connecting...")
|
||||
elif state == ConnectionState.RECONNECTING:
|
||||
print("WebSocket is reconnecting...")
|
||||
elif state == ConnectionState.DISCONNECTED:
|
||||
print("WebSocket is disconnected")
|
||||
elif state == ConnectionState.ERROR:
|
||||
print("WebSocket has error")
|
||||
```
|
||||
|
||||
### Ping/Pong Mechanism
|
||||
|
||||
OKX requires specific ping/pong format:
|
||||
|
||||
```python
|
||||
# OKX expects simple "ping" string (not JSON)
|
||||
# The WebSocket client handles this automatically:
|
||||
|
||||
# Send: "ping"
|
||||
# Receive: "pong"
|
||||
|
||||
# This is handled automatically by OKXWebSocketClient
|
||||
# Ping interval must be < 30 seconds to avoid disconnection
|
||||
```
|
||||
|
||||
## Error Handling and Troubleshooting
|
||||
|
||||
### Common Issues and Solutions
|
||||
|
||||
#### 1. Connection Failures
|
||||
|
||||
```python
|
||||
# Check connection status
|
||||
status = collector.get_status()
|
||||
if not status['websocket_connected']:
|
||||
print("WebSocket not connected")
|
||||
|
||||
# Check WebSocket state
|
||||
ws_state = status.get('websocket_state', 'unknown')
|
||||
|
||||
if ws_state == 'error':
|
||||
print("WebSocket in error state - will auto-restart")
|
||||
elif ws_state == 'reconnecting':
|
||||
print("WebSocket is reconnecting...")
|
||||
|
||||
# Manual restart if needed
|
||||
await collector.restart()
|
||||
```
|
||||
|
||||
#### 2. Ping/Pong Issues
|
||||
|
||||
```python
|
||||
# Monitor ping/pong status
|
||||
if 'websocket_stats' in status:
|
||||
ws_stats = status['websocket_stats']
|
||||
pings_sent = ws_stats.get('pings_sent', 0)
|
||||
pongs_received = ws_stats.get('pongs_received', 0)
|
||||
|
||||
if pings_sent > pongs_received + 3: # Allow some tolerance
|
||||
print("Ping/pong issue detected - connection may be stale")
|
||||
# Auto-restart will handle this
|
||||
```
|
||||
|
||||
#### 3. Data Validation Errors
|
||||
|
||||
```python
|
||||
# Monitor for validation errors
|
||||
errors = status.get('errors', 0)
|
||||
if errors > 0:
|
||||
print(f"Data validation errors detected: {errors}")
|
||||
|
||||
# Check logs for details:
|
||||
# - Malformed messages
|
||||
# - Missing required fields
|
||||
# - Invalid data types
|
||||
```
|
||||
|
||||
#### 4. Performance Issues
|
||||
|
||||
```python
|
||||
# Monitor message processing rate
|
||||
messages = status.get('messages_processed', 0)
|
||||
uptime = status.get('uptime_seconds', 1)
|
||||
rate = messages / uptime
|
||||
|
||||
if rate < 1.0: # Less than 1 message per second
|
||||
print("Low message rate - check:")
|
||||
print("- Network connectivity")
|
||||
print("- OKX API status")
|
||||
print("- Symbol activity")
|
||||
```
|
||||
|
||||
### Debug Mode
|
||||
|
||||
Enable debug logging for detailed information:
|
||||
|
||||
```python
|
||||
import os
|
||||
os.environ['LOG_LEVEL'] = 'DEBUG'
|
||||
|
||||
# Create collector with verbose logging
|
||||
collector = create_okx_collector(
|
||||
symbol='BTC-USDT',
|
||||
data_types=[DataType.TRADE, DataType.ORDERBOOK]
|
||||
)
|
||||
|
||||
await collector.start()
|
||||
|
||||
# Check logs in ./logs/ directory:
|
||||
# - okx_collector_btc_usdt_debug.log
|
||||
# - okx_collector_btc_usdt_info.log
|
||||
# - okx_collector_btc_usdt_error.log
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit Tests
|
||||
|
||||
Run the existing test scripts:
|
||||
|
||||
```bash
|
||||
# Test single collector
|
||||
python scripts/test_okx_collector.py single
|
||||
|
||||
# Test collector manager
|
||||
python scripts/test_okx_collector.py manager
|
||||
|
||||
# Test factory pattern
|
||||
python scripts/test_exchange_factory.py
|
||||
```
|
||||
|
||||
### Custom Testing
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from data.exchanges import create_okx_collector
|
||||
from data.base_collector import DataType
|
||||
|
||||
async def test_okx_collector():
|
||||
"""Test OKX collector functionality."""
|
||||
|
||||
# Test data collection
|
||||
message_count = 0
|
||||
error_count = 0
|
||||
|
||||
def on_trade(data_point):
|
||||
nonlocal message_count
|
||||
message_count += 1
|
||||
print(f"Trade #{message_count}: {data_point.data.get('tradeId')}")
|
||||
|
||||
def on_error(error):
|
||||
nonlocal error_count
|
||||
error_count += 1
|
||||
print(f"Error #{error_count}: {error}")
|
||||
|
||||
# Create and configure collector
|
||||
collector = create_okx_collector(
|
||||
symbol='BTC-USDT',
|
||||
data_types=[DataType.TRADE],
|
||||
auto_restart=True
|
||||
)
|
||||
|
||||
collector.add_data_callback(DataType.TRADE, on_trade)
|
||||
|
||||
# Test lifecycle
|
||||
print("Starting collector...")
|
||||
await collector.start()
|
||||
|
||||
print("Collecting data for 30 seconds...")
|
||||
await asyncio.sleep(30)
|
||||
|
||||
print("Stopping collector...")
|
||||
await collector.stop()
|
||||
|
||||
# Check results
|
||||
status = collector.get_status()
|
||||
print(f"Final status: {status['status']}")
|
||||
print(f"Messages processed: {status.get('messages_processed', 0)}")
|
||||
print(f"Errors: {status.get('errors', 0)}")
|
||||
|
||||
assert message_count > 0, "No messages received"
|
||||
assert error_count == 0, f"Unexpected errors: {error_count}"
|
||||
|
||||
print("Test passed!")
|
||||
|
||||
# Run test
|
||||
asyncio.run(test_okx_collector())
|
||||
```
|
||||
|
||||
## Production Deployment
|
||||
|
||||
### Recommended Configuration
|
||||
|
||||
```python
|
||||
# Production-ready OKX collector setup
|
||||
import asyncio
|
||||
from data.collector_manager import CollectorManager
|
||||
from data.exchanges import create_okx_collector
|
||||
from data.base_collector import DataType
|
||||
|
||||
async def deploy_okx_production():
|
||||
"""Production deployment configuration."""
|
||||
|
||||
# Create manager with appropriate settings
|
||||
manager = CollectorManager(
|
||||
manager_name="okx_production",
|
||||
global_health_check_interval=30.0, # Check every 30 seconds
|
||||
restart_delay=10.0 # Wait 10 seconds between restarts
|
||||
)
|
||||
|
||||
# Production trading pairs
|
||||
trading_pairs = [
|
||||
'BTC-USDT', 'ETH-USDT', 'SOL-USDT',
|
||||
'DOGE-USDT', 'TON-USDT', 'UNI-USDT'
|
||||
]
|
||||
|
||||
# Create collectors with production settings
|
||||
for symbol in trading_pairs:
|
||||
collector = create_okx_collector(
|
||||
symbol=symbol,
|
||||
data_types=[DataType.TRADE, DataType.ORDERBOOK],
|
||||
auto_restart=True,
|
||||
health_check_interval=15.0, # More frequent health checks
|
||||
store_raw_data=False # Disable raw data storage in production
|
||||
)
|
||||
|
||||
manager.add_collector(collector)
|
||||
|
||||
# Start system
|
||||
await manager.start()
|
||||
|
||||
# Production monitoring loop
|
||||
try:
|
||||
while True:
|
||||
await asyncio.sleep(60) # Check every minute
|
||||
|
||||
status = manager.get_status()
|
||||
stats = status.get('statistics', {})
|
||||
|
||||
# Log production metrics
|
||||
print(f"=== Production Status ===")
|
||||
print(f"Running: {stats.get('running_collectors', 0)}/{len(trading_pairs)}")
|
||||
print(f"Failed: {stats.get('failed_collectors', 0)}")
|
||||
print(f"Total restarts: {stats.get('restarts_performed', 0)}")
|
||||
|
||||
# Alert on failures
|
||||
failed_count = stats.get('failed_collectors', 0)
|
||||
if failed_count > 0:
|
||||
print(f"ALERT: {failed_count} collectors failed!")
|
||||
# Implement alerting system here
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("Shutting down production system...")
|
||||
await manager.stop()
|
||||
print("Production system stopped")
|
||||
|
||||
# Deploy to production
|
||||
asyncio.run(deploy_okx_production())
|
||||
```
|
||||
|
||||
### Docker Deployment
|
||||
|
||||
```dockerfile
|
||||
# Dockerfile for OKX collector
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
COPY requirements.txt .
|
||||
RUN pip install -r requirements.txt
|
||||
|
||||
COPY . .
|
||||
|
||||
# Production command
|
||||
CMD ["python", "-m", "scripts.deploy_okx_production"]
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
# Production environment variables
|
||||
export LOG_LEVEL=INFO
|
||||
export OKX_ENV=production
|
||||
export HEALTH_CHECK_INTERVAL=30
|
||||
export AUTO_RESTART=true
|
||||
export STORE_RAW_DATA=false
|
||||
export DATABASE_URL=postgresql://user:pass@host:5432/db
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### OKXCollector Class
|
||||
|
||||
```python
|
||||
class OKXCollector(BaseDataCollector):
|
||||
def __init__(self,
|
||||
symbol: str,
|
||||
data_types: Optional[List[DataType]] = None,
|
||||
component_name: Optional[str] = None,
|
||||
auto_restart: bool = True,
|
||||
health_check_interval: float = 30.0,
|
||||
store_raw_data: bool = True):
|
||||
"""
|
||||
Initialize OKX collector.
|
||||
|
||||
Args:
|
||||
symbol: Trading symbol (e.g., 'BTC-USDT')
|
||||
data_types: Data types to collect (default: [TRADE, ORDERBOOK])
|
||||
component_name: Name for logging (default: auto-generated)
|
||||
auto_restart: Enable automatic restart on failures
|
||||
health_check_interval: Seconds between health checks
|
||||
store_raw_data: Whether to store raw OKX data
|
||||
"""
|
||||
```
|
||||
|
||||
### OKXWebSocketClient Class
|
||||
|
||||
```python
|
||||
class OKXWebSocketClient:
|
||||
def __init__(self,
|
||||
component_name: str = "okx_websocket",
|
||||
ping_interval: float = 25.0,
|
||||
pong_timeout: float = 10.0,
|
||||
max_reconnect_attempts: int = 5,
|
||||
reconnect_delay: float = 5.0):
|
||||
"""
|
||||
Initialize OKX WebSocket client.
|
||||
|
||||
Args:
|
||||
component_name: Name for logging
|
||||
ping_interval: Seconds between ping messages (must be < 30)
|
||||
pong_timeout: Seconds to wait for pong response
|
||||
max_reconnect_attempts: Maximum reconnection attempts
|
||||
reconnect_delay: Initial delay between reconnection attempts
|
||||
"""
|
||||
```
|
||||
|
||||
### Factory Functions
|
||||
|
||||
```python
|
||||
def create_okx_collector(symbol: str,
|
||||
data_types: Optional[List[DataType]] = None,
|
||||
**kwargs) -> BaseDataCollector:
|
||||
"""
|
||||
Create OKX collector using convenience function.
|
||||
|
||||
Args:
|
||||
symbol: Trading pair symbol
|
||||
data_types: Data types to collect
|
||||
**kwargs: Additional collector parameters
|
||||
|
||||
Returns:
|
||||
OKXCollector instance
|
||||
"""
|
||||
|
||||
def ExchangeFactory.create_collector(config: ExchangeCollectorConfig) -> BaseDataCollector:
|
||||
"""
|
||||
Create collector using factory pattern.
|
||||
|
||||
Args:
|
||||
config: Exchange collector configuration
|
||||
|
||||
Returns:
|
||||
Appropriate collector instance
|
||||
"""
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Support
|
||||
|
||||
For OKX collector issues:
|
||||
|
||||
1. **Check Status**: Use `get_status()` and `get_health_status()` methods
|
||||
2. **Review Logs**: Check logs in `./logs/` directory
|
||||
3. **Debug Mode**: Set `LOG_LEVEL=DEBUG` for detailed logging
|
||||
4. **Test Connection**: Run `scripts/test_okx_collector.py`
|
||||
5. **Verify Configuration**: Check `config/okx_config.json`
|
||||
|
||||
For more information, see the main [Data Collectors Documentation](data_collectors.md).
|
||||
Reference in New Issue
Block a user