TCPDashboard/docs/architecture.md

Architecture Components

1. Data Collector

Responsibility: OHLCV data collection and aggregation from exchanges

class DataCollector:
    def __init__(self):
        self.providers = {}  # Registry of data providers
        self.store_raw_data = False  # Optional raw data storage
        
    def register_provider(self, name: str, provider: DataProvider):
        """Register a new data provider"""
        
    def start_collection(self, symbols: List[str], timeframes: List[str]):
        """Start collecting OHLCV data for specified symbols and timeframes"""
        
    def process_raw_trades(self, raw_trades: List[dict]) -> dict:
        """Aggregate raw trades into OHLCV candles"""
        
    def store_ohlcv_data(self, ohlcv_data: dict):
        """Store OHLCV data in PostgreSQL market_data table"""
        
    def send_market_update(self, symbol: str, ohlcv_data: dict):
        """Send Redis signal with OHLCV update to active bots"""
        
    def store_raw_data_optional(self, raw_data: dict):
        """Optionally store raw data for detailed backtesting"""

2. Strategy Engine

Responsibility: Unified interface for all trading strategies

class BaseStrategy:
    def __init__(self, parameters: dict):
        self.parameters = parameters
        
    def process_data(self, data: pd.DataFrame) -> Signal:
        """Process market data and generate signals"""
        raise NotImplementedError
        
    def get_indicators(self) -> dict:
        """Return calculated indicators for plotting"""
        return {}

3. Bot Manager

Responsibility: Orchestrate bot execution and state management

class BotManager:
    def __init__(self):
        self.active_bots = {}
        self.config_path = "config/bots/"
        
    def load_bot_config(self, bot_id: int) -> dict:
        """Load bot configuration from JSON file"""
        
    def start_bot(self, bot_id: int):
        """Start a bot instance with crash recovery monitoring"""
        
    def stop_bot(self, bot_id: int):
        """Stop a bot instance and update database status"""
        
    def process_signal(self, bot_id: int, signal: Signal):
        """Process signal and make virtual trading decision"""
        
    def update_bot_heartbeat(self, bot_id: int):
        """Update bot heartbeat in database for monitoring"""
        
    def restart_crashed_bots(self):
        """Monitor and restart crashed bots (max 3 attempts/hour)"""
        
    def restore_active_bots_on_startup(self):
        """Restore active bot states after application restart"""

Communication Architecture

Redis Pub/Sub Patterns

# Real-time market data distribution
MARKET_DATA_CHANNEL = "market:{symbol}"           # OHLCV updates
BOT_SIGNALS_CHANNEL = "signals:{bot_id}"         # Trading decisions
BOT_STATUS_CHANNEL = "status:{bot_id}"           # Bot lifecycle events
SYSTEM_EVENTS_CHANNEL = "system:events"         # Global notifications

Time Aggregation Strategy

Candlestick Alignment

• Use RIGHT-ALIGNED timestamps (industry standard)
• 5-minute candle with timestamp 09:05:00 represents data from 09:00:01 to 09:05:00
• Timestamp = close time of the candle
• Aligns with major exchanges (Binance, OKX, Coinbase)

Aggregation Logic

def aggregate_to_timeframe(ticks: List[dict], timeframe: str) -> dict:
    """
    Aggregate tick data to specified timeframe
    timeframe: '1m', '5m', '15m', '1h', '4h', '1d'
    """
    # Convert timeframe to seconds
    interval_seconds = parse_timeframe(timeframe)
    
    # Group ticks by time intervals (right-aligned)
    for group in group_by_interval(ticks, interval_seconds):
        candle = {
            'timestamp': group.end_time,  # Right-aligned
            'open': group.first_price,
            'high': group.max_price,
            'low': group.min_price,
            'close': group.last_price,
            'volume': group.total_volume
        }
        yield candle

Backtesting Strategy

Vectorized Processing Approach

import pandas as pd
import numpy as np

def backtest_strategy_simple(strategy, market_data: pd.DataFrame, initial_balance: float = 10000):
    """
    Simple vectorized backtesting using pandas operations
    
    Parameters:
    - strategy: Strategy instance with process_data method
    - market_data: DataFrame with OHLCV data
    - initial_balance: Starting portfolio value
    
    Returns:
    - Portfolio performance metrics and trade history
    """
    
    # Calculate all signals at once using vectorized operations
    signals = []
    portfolio_value = []
    current_balance = initial_balance
    position = 0
    
    for idx, row in market_data.iterrows():
        # Get signal from strategy
        signal = strategy.process_data(market_data.iloc[:idx+1])
        
        # Simulate trade execution
        if signal.action == 'buy' and position == 0:
            position = current_balance / row['close']
            current_balance = 0
            
        elif signal.action == 'sell' and position > 0:
            current_balance = position * row['close'] * 0.999  # 0.1% fee
            position = 0
            
        # Track portfolio value
        total_value = current_balance + (position * row['close'])
        portfolio_value.append(total_value)
        signals.append(signal)
    
    return {
        'final_value': portfolio_value[-1],
        'total_return': (portfolio_value[-1] / initial_balance - 1) * 100,
        'signals': signals,
        'portfolio_progression': portfolio_value
    }

def calculate_performance_metrics(portfolio_values: List[float]) -> dict:
    """Calculate standard performance metrics"""
    returns = pd.Series(portfolio_values).pct_change().dropna()
    
    return {
        'sharpe_ratio': returns.mean() / returns.std() if returns.std() > 0 else 0,
        'max_drawdown': (pd.Series(portfolio_values).cummax() - pd.Series(portfolio_values)).max(),
        'win_rate': (returns > 0).mean(),
        'total_trades': len(returns)
    }

Optimization Techniques

1. Vectorized Operations: Use pandas for bulk data processing
2. Efficient Indexing: Pre-calculate indicators where possible
3. Memory Management: Process data in chunks for large datasets
4. Simple Parallelization: Run multiple strategy tests independently

Key Design Principles

1. OHLCV-First Data Strategy: Primary focus on aggregated candle data, optional raw data storage
2. Signal Tracking: All trading signals recorded in database for analysis and debugging
3. JSON Configuration: Strategy parameters and bot configs in JSON for rapid testing
4. Real-time State Management: Bot states updated via Redis and PostgreSQL for monitoring
5. Crash Recovery: Automatic bot restart and application state recovery
6. Virtual Trading: Simulation-first approach with fee modeling
7. Simplified Architecture: Monolithic design with clear component boundaries for future scaling

Database Architecture

Core Tables

• market_data: OHLCV candles for bot operations and backtesting (primary table)
• bots: Bot instances with JSON config references and status tracking
• signals: Trading decisions with confidence scores and indicator values
• trades: Virtual trade execution records with P&L tracking
• bot_performance: Portfolio snapshots for performance visualization

Optional Tables

• raw_trades: Raw tick data for advanced backtesting (partitioned by month)

Data Access Patterns

• Real-time: Bots read recent OHLCV data via indexes on (symbol, timeframe, timestamp)
• Historical: Dashboard queries aggregated performance data for charts
• Backtesting: Sequential access to historical OHLCV data by date range