vasily/TCPDashboard
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
TCPDashboard/docs/exchanges/okx_collector.md
Vasily.onl fa63e7eb2e documentation
2025-05-31 20:55:52 +08:00

945 lines
26 KiB
Markdown
Raw Blame History

# OKX Data Collector Documentation
## Overview
The OKX Data Collector provides real-time market data collection from OKX exchange using WebSocket API. It's built on the modular exchange architecture and provides robust connection management, automatic reconnection, health monitoring, and comprehensive data processing.
## Features
### 🎯 **OKX-Specific Features**
- **Real-time Data**: Live trades, orderbook, and ticker data
- **Single Pair Focus**: Each collector handles one trading pair for better isolation
- **Ping/Pong Management**: OKX-specific keepalive mechanism with proper format
- **Raw Data Storage**: Optional storage of raw OKX messages for debugging
- **Connection Resilience**: Robust reconnection logic for OKX WebSocket
### 📊 **Supported Data Types**
- **Trades**: Real-time trade executions (`trades` channel)
- **Orderbook**: 5-level order book depth (`books5` channel)
- **Ticker**: 24h ticker statistics (`tickers` channel)
- **Future**: Candle data support planned
### 🔧 **Configuration Options**
- Auto-restart on failures
- Health check intervals
- Raw data storage toggle
- Custom ping/pong timing
- Reconnection attempts configuration
## Quick Start
### 1. Using Factory Pattern (Recommended)
```python
import asyncio
from data.exchanges import create_okx_collector
from data.base_collector import DataType
async def main():
# Create OKX collector using convenience function
collector = create_okx_collector(
symbol='BTC-USDT',
data_types=[DataType.TRADE, DataType.ORDERBOOK],
auto_restart=True,
health_check_interval=30.0,
store_raw_data=True
)
# Add data callbacks
def on_trade(data_point):
trade = data_point.data
print(f"Trade: {trade['side']} {trade['sz']} @ {trade['px']} (ID: {trade['tradeId']})")
def on_orderbook(data_point):
book = data_point.data
if book.get('bids') and book.get('asks'):
best_bid = book['bids'][0]
best_ask = book['asks'][0]
print(f"Orderbook: Bid {best_bid[0]}@{best_bid[1]} Ask {best_ask[0]}@{best_ask[1]}")
collector.add_data_callback(DataType.TRADE, on_trade)
collector.add_data_callback(DataType.ORDERBOOK, on_orderbook)
# Start collector
await collector.start()
# Run for 60 seconds
await asyncio.sleep(60)
# Stop gracefully
await collector.stop()
asyncio.run(main())
```
### 2. Direct OKX Collector Usage
```python
import asyncio
from data.exchanges.okx import OKXCollector
from data.base_collector import DataType
async def main():
# Create collector directly
collector = OKXCollector(
symbol='ETH-USDT',
data_types=[DataType.TRADE, DataType.ORDERBOOK],
component_name='eth_collector',
auto_restart=True,
health_check_interval=30.0,
store_raw_data=True
)
# Add callbacks
def on_data(data_point):
print(f"{data_point.data_type.value}: {data_point.symbol} - {data_point.timestamp}")
collector.add_data_callback(DataType.TRADE, on_data)
collector.add_data_callback(DataType.ORDERBOOK, on_data)
# Start and monitor
await collector.start()
# Monitor status
for i in range(12): # 60 seconds total
await asyncio.sleep(5)
status = collector.get_status()
print(f"Status: {status['status']} - Messages: {status.get('messages_processed', 0)}")
await collector.stop()
asyncio.run(main())
```
### 3. Multiple OKX Collectors with Manager
```python
import asyncio
from data.collector_manager import CollectorManager
from data.exchanges import create_okx_collector
from data.base_collector import DataType
async def main():
# Create manager
manager = CollectorManager(
manager_name="okx_trading_system",
global_health_check_interval=30.0
)
# Create multiple OKX collectors
symbols = ['BTC-USDT', 'ETH-USDT', 'SOL-USDT']
for symbol in symbols:
collector = create_okx_collector(
symbol=symbol,
data_types=[DataType.TRADE, DataType.ORDERBOOK],
auto_restart=True
)
manager.add_collector(collector)
# Start manager
await manager.start()
# Monitor all collectors
while True:
status = manager.get_status()
stats = status.get('statistics', {})
print(f"=== OKX Collectors Status ===")
print(f"Running: {stats.get('running_collectors', 0)}")
print(f"Failed: {stats.get('failed_collectors', 0)}")
print(f"Total messages: {stats.get('total_messages', 0)}")
# Individual collector status
for collector_name in manager.list_collectors():
collector_status = manager.get_collector_status(collector_name)
if collector_status:
info = collector_status.get('status', {})
print(f" {collector_name}: {info.get('status')} - "
f"Messages: {info.get('messages_processed', 0)}")
await asyncio.sleep(15)
asyncio.run(main())
```
## Configuration
### 1. JSON Configuration File
The system uses `config/okx_config.json` for configuration:
```json
{
"exchange": "okx",
"connection": {
"public_ws_url": "wss://ws.okx.com:8443/ws/v5/public",
"private_ws_url": "wss://ws.okx.com:8443/ws/v5/private",
"ping_interval": 25.0,
"pong_timeout": 10.0,
"max_reconnect_attempts": 5,
"reconnect_delay": 5.0
},
"data_collection": {
"store_raw_data": true,
"health_check_interval": 30.0,
"auto_restart": true,
"buffer_size": 1000
},
"factory": {
"use_factory_pattern": true,
"default_data_types": ["trade", "orderbook"],
"batch_create": true
},
"trading_pairs": [
{
"symbol": "BTC-USDT",
"enabled": true,
"data_types": ["trade", "orderbook"],
"channels": {
"trades": "trades",
"orderbook": "books5",
"ticker": "tickers"
}
},
{
"symbol": "ETH-USDT",
"enabled": true,
"data_types": ["trade", "orderbook"],
"channels": {
"trades": "trades",
"orderbook": "books5",
"ticker": "tickers"
}
}
],
"logging": {
"component_name_template": "okx_collector_{symbol}",
"log_level": "INFO",
"verbose": false
},
"database": {
"store_processed_data": true,
"store_raw_data": true,
"batch_size": 100,
"flush_interval": 5.0
},
"monitoring": {
"enable_health_checks": true,
"health_check_interval": 30.0,
"alert_on_connection_loss": true,
"max_consecutive_errors": 5
}
}
```
### 2. Programmatic Configuration
```python
from data.exchanges.okx import OKXCollector
from data.base_collector import DataType
# Custom configuration
collector = OKXCollector(
symbol='BTC-USDT',
data_types=[DataType.TRADE, DataType.ORDERBOOK],
component_name='custom_btc_collector',
auto_restart=True,
health_check_interval=15.0, # Check every 15 seconds
store_raw_data=True # Store raw OKX messages
)
```
### 3. Factory Configuration
```python
from data.exchanges import ExchangeFactory, ExchangeCollectorConfig
from data.base_collector import DataType
config = ExchangeCollectorConfig(
exchange='okx',
symbol='ETH-USDT',
data_types=[DataType.TRADE, DataType.ORDERBOOK],
auto_restart=True,
health_check_interval=30.0,
store_raw_data=True,
custom_params={
'ping_interval': 20.0, # Custom ping interval
'max_reconnect_attempts': 10, # More reconnection attempts
'pong_timeout': 15.0 # Longer pong timeout
}
)
collector = ExchangeFactory.create_collector(config)
```
## Data Processing
### OKX Message Formats
#### Trade Data
```python
# Raw OKX trade message
{
"arg": {
"channel": "trades",
"instId": "BTC-USDT"
},
"data": [
{
"instId": "BTC-USDT",
"tradeId": "12345678",
"px": "50000.5", # Price
"sz": "0.001", # Size
"side": "buy", # Side (buy/sell)
"ts": "1697123456789" # Timestamp (ms)
}
]
}
# Processed MarketDataPoint
MarketDataPoint(
exchange="okx",
symbol="BTC-USDT",
timestamp=datetime(2023, 10, 12, 15, 30, 56, tzinfo=timezone.utc),
data_type=DataType.TRADE,
data={
"instId": "BTC-USDT",
"tradeId": "12345678",
"px": "50000.5",
"sz": "0.001",
"side": "buy",
"ts": "1697123456789"
}
)
```
#### Orderbook Data
```python
# Raw OKX orderbook message (books5)
{
"arg": {
"channel": "books5",
"instId": "BTC-USDT"
},
"data": [
{
"asks": [
["50001.0", "0.5", "0", "3"], # [price, size, liquidated, orders]
["50002.0", "1.0", "0", "5"]
],
"bids": [
["50000.0", "0.8", "0", "2"],
["49999.0", "1.2", "0", "4"]
],
"ts": "1697123456789",
"checksum": "123456789"
}
]
}
# Usage in callback
def on_orderbook(data_point):
book = data_point.data
if book.get('bids') and book.get('asks'):
best_bid = book['bids'][0]
best_ask = book['asks'][0]
spread = float(best_ask[0]) - float(best_bid[0])
print(f"Spread: ${spread:.2f}")
```
#### Ticker Data
```python
# Raw OKX ticker message
{
"arg": {
"channel": "tickers",
"instId": "BTC-USDT"
},
"data": [
{
"instType": "SPOT",
"instId": "BTC-USDT",
"last": "50000.5", # Last price
"lastSz": "0.001", # Last size
"askPx": "50001.0", # Best ask price
"askSz": "0.5", # Best ask size
"bidPx": "50000.0", # Best bid price
"bidSz": "0.8", # Best bid size
"open24h": "49500.0", # 24h open
"high24h": "50500.0", # 24h high
"low24h": "49000.0", # 24h low
"vol24h": "1234.567", # 24h volume
"ts": "1697123456789"
}
]
}
```
### Data Validation
The OKX collector includes comprehensive data validation:
```python
# Automatic validation in collector
class OKXCollector(BaseDataCollector):
async def _process_data_item(self, channel: str, data_item: Dict[str, Any]):
# Validate message structure
if not isinstance(data_item, dict):
self.logger.warning("Invalid data item type")
return None
# Validate required fields based on channel
if channel == "trades":
required_fields = ['tradeId', 'px', 'sz', 'side', 'ts']
elif channel == "books5":
required_fields = ['bids', 'asks', 'ts']
elif channel == "tickers":
required_fields = ['last', 'ts']
else:
self.logger.warning(f"Unknown channel: {channel}")
return None
# Check required fields
for field in required_fields:
if field not in data_item:
self.logger.warning(f"Missing required field '{field}' in {channel} data")
return None
# Process and return validated data
return await self._create_market_data_point(channel, data_item)
```
## Monitoring and Status
### Status Information
```python
# Get comprehensive status
status = collector.get_status()
print(f"Exchange: {status['exchange']}") # 'okx'
print(f"Symbol: {status['symbol']}") # 'BTC-USDT'
print(f"Status: {status['status']}") # 'running'
print(f"WebSocket Connected: {status['websocket_connected']}") # True/False
print(f"WebSocket State: {status['websocket_state']}") # 'connected'
print(f"Messages Processed: {status['messages_processed']}") # Integer
print(f"Errors: {status['errors']}") # Integer
print(f"Last Trade ID: {status['last_trade_id']}") # String or None
# WebSocket statistics
if 'websocket_stats' in status:
ws_stats = status['websocket_stats']
print(f"Messages Received: {ws_stats['messages_received']}")
print(f"Messages Sent: {ws_stats['messages_sent']}")
print(f"Pings Sent: {ws_stats['pings_sent']}")
print(f"Pongs Received: {ws_stats['pongs_received']}")
print(f"Reconnections: {ws_stats['reconnections']}")
```
### Health Monitoring
```python
# Get health status
health = collector.get_health_status()
print(f"Is Healthy: {health['is_healthy']}") # True/False
print(f"Issues: {health['issues']}") # List of issues
print(f"Last Heartbeat: {health['last_heartbeat']}") # ISO timestamp
print(f"Last Data: {health['last_data_received']}") # ISO timestamp
print(f"Should Be Running: {health['should_be_running']}") # True/False
print(f"Is Running: {health['is_running']}") # True/False
# Auto-restart status
if not health['is_healthy']:
print("Collector is unhealthy - auto-restart will trigger")
for issue in health['issues']:
print(f" Issue: {issue}")
```
### Performance Monitoring
```python
import time
async def monitor_performance():
collector = create_okx_collector('BTC-USDT', [DataType.TRADE])
await collector.start()
start_time = time.time()
last_message_count = 0
while True:
await asyncio.sleep(10) # Check every 10 seconds
status = collector.get_status()
current_messages = status.get('messages_processed', 0)
# Calculate message rate
elapsed = time.time() - start_time
messages_per_second = current_messages / elapsed if elapsed > 0 else 0
# Calculate recent rate
recent_messages = current_messages - last_message_count
recent_rate = recent_messages / 10 # per second over last 10 seconds
print(f"=== Performance Stats ===")
print(f"Total Messages: {current_messages}")
print(f"Average Rate: {messages_per_second:.2f} msg/sec")
print(f"Recent Rate: {recent_rate:.2f} msg/sec")
print(f"Errors: {status.get('errors', 0)}")
print(f"WebSocket State: {status.get('websocket_state', 'unknown')}")
last_message_count = current_messages
# Run performance monitoring
asyncio.run(monitor_performance())
```
## WebSocket Connection Details
### OKX WebSocket Client
The OKX implementation includes a specialized WebSocket client:
```python
from data.exchanges.okx import OKXWebSocketClient, OKXSubscription, OKXChannelType
# Create WebSocket client directly (usually handled by collector)
ws_client = OKXWebSocketClient(
component_name='okx_ws_btc',
ping_interval=25.0, # Must be < 30 seconds for OKX
pong_timeout=10.0,
max_reconnect_attempts=5,
reconnect_delay=5.0
)
# Connect to OKX
await ws_client.connect(use_public=True)
# Create subscriptions
subscriptions = [
OKXSubscription(
channel=OKXChannelType.TRADES.value,
inst_id='BTC-USDT',
enabled=True
),
OKXSubscription(
channel=OKXChannelType.BOOKS5.value,
inst_id='BTC-USDT',
enabled=True
)
]
# Subscribe to channels
await ws_client.subscribe(subscriptions)
# Add message callback
def on_message(message):
print(f"Received: {message}")
ws_client.add_message_callback(on_message)
# WebSocket will handle messages automatically
await asyncio.sleep(60)
# Disconnect
await ws_client.disconnect()
```
### Connection States
The WebSocket client tracks connection states:
```python
from data.exchanges.okx.websocket import ConnectionState
# Check connection state
state = ws_client.connection_state
if state == ConnectionState.CONNECTED:
print("WebSocket is connected and ready")
elif state == ConnectionState.CONNECTING:
print("WebSocket is connecting...")
elif state == ConnectionState.RECONNECTING:
print("WebSocket is reconnecting...")
elif state == ConnectionState.DISCONNECTED:
print("WebSocket is disconnected")
elif state == ConnectionState.ERROR:
print("WebSocket has error")
```
### Ping/Pong Mechanism
OKX requires specific ping/pong format:
```python
# OKX expects simple "ping" string (not JSON)
# The WebSocket client handles this automatically:
# Send: "ping"
# Receive: "pong"
# This is handled automatically by OKXWebSocketClient
# Ping interval must be < 30 seconds to avoid disconnection
```
## Error Handling and Troubleshooting
### Common Issues and Solutions
#### 1. Connection Failures
```python
# Check connection status
status = collector.get_status()
if not status['websocket_connected']:
print("WebSocket not connected")
# Check WebSocket state
ws_state = status.get('websocket_state', 'unknown')
if ws_state == 'error':
print("WebSocket in error state - will auto-restart")
elif ws_state == 'reconnecting':
print("WebSocket is reconnecting...")
# Manual restart if needed
await collector.restart()
```
#### 2. Ping/Pong Issues
```python
# Monitor ping/pong status
if 'websocket_stats' in status:
ws_stats = status['websocket_stats']
pings_sent = ws_stats.get('pings_sent', 0)
pongs_received = ws_stats.get('pongs_received', 0)
if pings_sent > pongs_received + 3: # Allow some tolerance
print("Ping/pong issue detected - connection may be stale")
# Auto-restart will handle this
```
#### 3. Data Validation Errors
```python
# Monitor for validation errors
errors = status.get('errors', 0)
if errors > 0:
print(f"Data validation errors detected: {errors}")
# Check logs for details:
# - Malformed messages
# - Missing required fields
# - Invalid data types
```
#### 4. Performance Issues
```python
# Monitor message processing rate
messages = status.get('messages_processed', 0)
uptime = status.get('uptime_seconds', 1)
rate = messages / uptime
if rate < 1.0: # Less than 1 message per second
print("Low message rate - check:")
print("- Network connectivity")
print("- OKX API status")
print("- Symbol activity")
```
### Debug Mode
Enable debug logging for detailed information:
```python
import os
os.environ['LOG_LEVEL'] = 'DEBUG'
# Create collector with verbose logging
collector = create_okx_collector(
symbol='BTC-USDT',
data_types=[DataType.TRADE, DataType.ORDERBOOK]
)
await collector.start()
# Check logs in ./logs/ directory:
# - okx_collector_btc_usdt_debug.log
# - okx_collector_btc_usdt_info.log
# - okx_collector_btc_usdt_error.log
```
## Testing
### Unit Tests
Run the existing test scripts:
```bash
# Test single collector
python scripts/test_okx_collector.py single
# Test collector manager
python scripts/test_okx_collector.py manager
# Test factory pattern
python scripts/test_exchange_factory.py
```
### Custom Testing
```python
import asyncio
from data.exchanges import create_okx_collector
from data.base_collector import DataType
async def test_okx_collector():
"""Test OKX collector functionality."""
# Test data collection
message_count = 0
error_count = 0
def on_trade(data_point):
nonlocal message_count
message_count += 1
print(f"Trade #{message_count}: {data_point.data.get('tradeId')}")
def on_error(error):
nonlocal error_count
error_count += 1
print(f"Error #{error_count}: {error}")
# Create and configure collector
collector = create_okx_collector(
symbol='BTC-USDT',
data_types=[DataType.TRADE],
auto_restart=True
)
collector.add_data_callback(DataType.TRADE, on_trade)
# Test lifecycle
print("Starting collector...")
await collector.start()
print("Collecting data for 30 seconds...")
await asyncio.sleep(30)
print("Stopping collector...")
await collector.stop()
# Check results
status = collector.get_status()
print(f"Final status: {status['status']}")
print(f"Messages processed: {status.get('messages_processed', 0)}")
print(f"Errors: {status.get('errors', 0)}")
assert message_count > 0, "No messages received"
assert error_count == 0, f"Unexpected errors: {error_count}"
print("Test passed!")
# Run test
asyncio.run(test_okx_collector())
```
## Production Deployment
### Recommended Configuration
```python
# Production-ready OKX collector setup
import asyncio
from data.collector_manager import CollectorManager
from data.exchanges import create_okx_collector
from data.base_collector import DataType
async def deploy_okx_production():
"""Production deployment configuration."""
# Create manager with appropriate settings
manager = CollectorManager(
manager_name="okx_production",
global_health_check_interval=30.0, # Check every 30 seconds
restart_delay=10.0 # Wait 10 seconds between restarts
)
# Production trading pairs
trading_pairs = [
'BTC-USDT', 'ETH-USDT', 'SOL-USDT',
'DOGE-USDT', 'TON-USDT', 'UNI-USDT'
]
# Create collectors with production settings
for symbol in trading_pairs:
collector = create_okx_collector(
symbol=symbol,
data_types=[DataType.TRADE, DataType.ORDERBOOK],
auto_restart=True,
health_check_interval=15.0, # More frequent health checks
store_raw_data=False # Disable raw data storage in production
)
manager.add_collector(collector)
# Start system
await manager.start()
# Production monitoring loop
try:
while True:
await asyncio.sleep(60) # Check every minute
status = manager.get_status()
stats = status.get('statistics', {})
# Log production metrics
print(f"=== Production Status ===")
print(f"Running: {stats.get('running_collectors', 0)}/{len(trading_pairs)}")
print(f"Failed: {stats.get('failed_collectors', 0)}")
print(f"Total restarts: {stats.get('restarts_performed', 0)}")
# Alert on failures
failed_count = stats.get('failed_collectors', 0)
if failed_count > 0:
print(f"ALERT: {failed_count} collectors failed!")
# Implement alerting system here
except KeyboardInterrupt:
print("Shutting down production system...")
await manager.stop()
print("Production system stopped")
# Deploy to production
asyncio.run(deploy_okx_production())
```
### Docker Deployment
```dockerfile
# Dockerfile for OKX collector
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
# Production command
CMD ["python", "-m", "scripts.deploy_okx_production"]
```
### Environment Variables
```bash
# Production environment variables
export LOG_LEVEL=INFO
export OKX_ENV=production
export HEALTH_CHECK_INTERVAL=30
export AUTO_RESTART=true
export STORE_RAW_DATA=false
export DATABASE_URL=postgresql://user:pass@host:5432/db
```
## API Reference
### OKXCollector Class
```python
class OKXCollector(BaseDataCollector):
def __init__(self,
symbol: str,
data_types: Optional[List[DataType]] = None,
component_name: Optional[str] = None,
auto_restart: bool = True,
health_check_interval: float = 30.0,
store_raw_data: bool = True):
"""
Initialize OKX collector.
Args:
symbol: Trading symbol (e.g., 'BTC-USDT')
data_types: Data types to collect (default: [TRADE, ORDERBOOK])
component_name: Name for logging (default: auto-generated)
auto_restart: Enable automatic restart on failures
health_check_interval: Seconds between health checks
store_raw_data: Whether to store raw OKX data
"""
```
### OKXWebSocketClient Class
```python
class OKXWebSocketClient:
def __init__(self,
component_name: str = "okx_websocket",
ping_interval: float = 25.0,
pong_timeout: float = 10.0,
max_reconnect_attempts: int = 5,
reconnect_delay: float = 5.0):
"""
Initialize OKX WebSocket client.
Args:
component_name: Name for logging
ping_interval: Seconds between ping messages (must be < 30)
pong_timeout: Seconds to wait for pong response
max_reconnect_attempts: Maximum reconnection attempts
reconnect_delay: Initial delay between reconnection attempts
"""
```
### Factory Functions
```python
def create_okx_collector(symbol: str,
data_types: Optional[List[DataType]] = None,
**kwargs) -> BaseDataCollector:
"""
Create OKX collector using convenience function.
Args:
symbol: Trading pair symbol
data_types: Data types to collect
**kwargs: Additional collector parameters
Returns:
OKXCollector instance
"""
def ExchangeFactory.create_collector(config: ExchangeCollectorConfig) -> BaseDataCollector:
"""
Create collector using factory pattern.
Args:
config: Exchange collector configuration
Returns:
Appropriate collector instance
"""
```
---
## Support
For OKX collector issues:
1. **Check Status**: Use `get_status()` and `get_health_status()` methods
2. **Review Logs**: Check logs in `./logs/` directory
3. **Debug Mode**: Set `LOG_LEVEL=DEBUG` for detailed logging
4. **Test Connection**: Run `scripts/test_okx_collector.py`
5. **Verify Configuration**: Check `config/okx_config.json`
For more information, see the main [Data Collectors Documentation](data_collectors.md).
Reference in New Issue View Git Blame Copy Permalink