TCPDashboard/docs/components/charts/bot-integration.md

# Bot Integration with Chart Signal Layers

The Chart Layers System provides seamless integration with the bot management system, allowing real-time visualization of bot signals, trades, and performance data directly on charts.

## Table of Contents

- [Overview](#overview)
- [Architecture](#architecture)
- [Quick Start](#quick-start)
- [Bot Data Service](#bot-data-service)
- [Signal Layer Integration](#signal-layer-integration)
- [Enhanced Bot Layers](#enhanced-bot-layers)
- [Multi-Bot Visualization](#multi-bot-visualization)
- [Configuration Options](#configuration-options)
- [Examples](#examples)
- [Best Practices](#best-practices)
- [Troubleshooting](#troubleshooting)

## Overview

The bot integration system provides automatic data fetching and visualization of:

- **Trading Signals**: Buy/sell/hold signals from active bots
- **Trade Executions**: Entry/exit points with P&L information
- **Bot Performance**: Real-time performance metrics and analytics
- **Strategy Comparison**: Side-by-side strategy analysis
- **Multi-Bot Views**: Aggregate views across multiple bots

### Key Features

- **Automatic Data Fetching**: No manual data queries required
- **Real-time Updates**: Charts update with live bot data
- **Database Integration**: Direct connection to bot management system
- **Advanced Filtering**: Filter by bot, strategy, symbol, timeframe
- **Performance Analytics**: Built-in performance calculation
- **Error Handling**: Graceful handling of database errors

## Architecture

```
components/charts/layers/
├── bot_integration.py         # Core bot data services
├── bot_enhanced_layers.py     # Enhanced chart layers with bot integration
└── signals.py                 # Base signal layers

Bot Integration Components:
├── BotFilterConfig           # Configuration for bot filtering
├── BotDataService           # Database operations for bot data
├── BotSignalLayerIntegration # Chart-specific integration utilities
├── BotIntegratedSignalLayer # Auto-fetching signal layer
├── BotIntegratedTradeLayer  # Auto-fetching trade layer
└── BotMultiLayerIntegration # Multi-bot layer management
```

## Quick Start

### Basic Bot Signal Visualization

```python
from components.charts.layers import create_bot_signal_layer

# Create a bot-integrated signal layer for BTCUSDT
signal_layer = create_bot_signal_layer(
    symbol='BTCUSDT',
    active_only=True,
    confidence_threshold=0.5,
    time_window_days=7
)

# Add to chart
fig = go.Figure()
fig = signal_layer.render(fig, market_data, symbol='BTCUSDT')
```

### Complete Bot Visualization Setup

```python
from components.charts.layers import create_complete_bot_layers

# Create complete bot layer set for a symbol
result = create_complete_bot_layers(
    symbol='BTCUSDT',
    timeframe='1h',
    active_only=True,
    time_window_days=7
)

if result['success']:
    signal_layer = result['layers']['signals']
    trade_layer = result['layers']['trades']
    
    # Add to chart
    fig = signal_layer.render(fig, market_data, symbol='BTCUSDT')
    fig = trade_layer.render(fig, market_data, symbol='BTCUSDT')
```

## Bot Data Service

The `BotDataService` provides the core interface for fetching bot-related data from the database.

### Basic Usage

```python
from components.charts.layers.bot_integration import BotDataService, BotFilterConfig

# Initialize service
service = BotDataService()

# Create filter configuration
bot_filter = BotFilterConfig(
    symbols=['BTCUSDT'],
    strategies=['momentum', 'ema_crossover'],
    active_only=True
)

# Fetch bot data
bots_df = service.get_bots(bot_filter)
signals_df = service.get_signals_for_bots(
    bot_ids=bots_df['id'].tolist(),
    start_time=datetime.now() - timedelta(days=7),
    end_time=datetime.now(),
    min_confidence=0.3
)
```

### Available Methods

| Method | Description | Parameters |
|--------|-------------|------------|
| `get_bots()` | Fetch bot information | `filter_config: BotFilterConfig` |
| `get_signals_for_bots()` | Fetch signals from bots | `bot_ids, start_time, end_time, signal_types, min_confidence` |
| `get_trades_for_bots()` | Fetch trades from bots | `bot_ids, start_time, end_time, sides` |
| `get_bot_performance()` | Fetch performance data | `bot_ids, start_time, end_time` |

### BotFilterConfig Options

```python
@dataclass
class BotFilterConfig:
    bot_ids: Optional[List[int]] = None        # Specific bot IDs
    bot_names: Optional[List[str]] = None      # Specific bot names
    strategies: Optional[List[str]] = None     # Strategy filter
    symbols: Optional[List[str]] = None        # Symbol filter
    statuses: Optional[List[str]] = None       # Bot status filter
    date_range: Optional[Tuple[datetime, datetime]] = None
    active_only: bool = False                  # Only active bots
```

## Signal Layer Integration

The `BotSignalLayerIntegration` provides chart-specific utilities for integrating bot data with chart layers.

### Chart-Specific Signal Fetching

```python
from components.charts.layers.bot_integration import BotSignalLayerIntegration

integration = BotSignalLayerIntegration()

# Get signals for specific chart context
signals_df = integration.get_signals_for_chart(
    symbol='BTCUSDT',
    timeframe='1h',
    bot_filter=BotFilterConfig(active_only=True),
    time_range=(start_time, end_time),
    signal_types=['buy', 'sell'],
    min_confidence=0.5
)

# Get trades for chart context
trades_df = integration.get_trades_for_chart(
    symbol='BTCUSDT',
    timeframe='1h',
    bot_filter=BotFilterConfig(strategies=['momentum']),
    time_range=(start_time, end_time)
)

# Get bot summary statistics
stats = integration.get_bot_summary_stats(bot_ids=[1, 2, 3])
```

### Performance Analytics

```python
# Get comprehensive performance summary
performance = get_bot_performance_summary(
    bot_id=1,  # Specific bot or None for all
    days_back=30
)

print(f"Total trades: {performance['trade_count']}")
print(f"Win rate: {performance['win_rate']:.1f}%")
print(f"Total P&L: ${performance['bot_stats']['total_pnl']:.2f}")
```

## Enhanced Bot Layers

Enhanced layers provide automatic data fetching and bot-specific visualization features.

### BotIntegratedSignalLayer

```python
from components.charts.layers import BotIntegratedSignalLayer, BotSignalLayerConfig

# Configure bot-integrated signal layer
config = BotSignalLayerConfig(
    name="BTCUSDT Bot Signals",
    auto_fetch_data=True,              # Automatically fetch from database
    time_window_days=7,                # Look back 7 days
    active_bots_only=True,             # Only active bots
    include_bot_info=True,             # Include bot info in hover
    group_by_strategy=True,            # Group signals by strategy
    confidence_threshold=0.3,          # Minimum confidence
    signal_types=['buy', 'sell']       # Signal types to show
)

layer = BotIntegratedSignalLayer(config)

# Render automatically fetches data
fig = layer.render(fig, market_data, symbol='BTCUSDT')
```

### BotIntegratedTradeLayer

```python
from components.charts.layers import BotIntegratedTradeLayer, BotTradeLayerConfig

config = BotTradeLayerConfig(
    name="BTCUSDT Bot Trades",
    auto_fetch_data=True,
    time_window_days=7,
    show_pnl=True,                     # Show profit/loss
    show_trade_lines=True,             # Connect entry/exit
    include_bot_info=True,             # Bot info in hover
    group_by_strategy=False
)

layer = BotIntegratedTradeLayer(config)
fig = layer.render(fig, market_data, symbol='BTCUSDT')
```

## Multi-Bot Visualization

### Strategy Comparison

```python
from components.charts.layers import bot_multi_layer

# Compare multiple strategies on the same symbol
result = bot_multi_layer.create_strategy_comparison_layers(
    symbol='BTCUSDT',
    strategies=['momentum', 'ema_crossover', 'mean_reversion'],
    timeframe='1h',
    time_window_days=14
)

if result['success']:
    for strategy in result['strategies']:
        signal_layer = result['layers'][f"{strategy}_signals"]
        trade_layer = result['layers'][f"{strategy}_trades"]
        
        fig = signal_layer.render(fig, market_data, symbol='BTCUSDT')
        fig = trade_layer.render(fig, market_data, symbol='BTCUSDT')
```

### Multi-Symbol Bot View

```python
# Create bot layers for multiple symbols
symbols = ['BTCUSDT', 'ETHUSDT', 'ADAUSDT']

for symbol in symbols:
    bot_layers = create_complete_bot_layers(
        symbol=symbol,
        active_only=True,
        time_window_days=7
    )
    
    if bot_layers['success']:
        # Add layers to respective charts
        signal_layer = bot_layers['layers']['signals']
        # ... render on symbol-specific chart
```

## Configuration Options

### Auto-Fetch Configuration

```python
# Disable auto-fetch for manual data control
config = BotSignalLayerConfig(
    name="Manual Bot Signals",
    auto_fetch_data=False,             # Disable auto-fetch
    active_bots_only=True
)

layer = BotIntegratedSignalLayer(config)

# Manually provide signal data
manual_signals = get_signals_from_api()
fig = layer.render(fig, market_data, signals=manual_signals)
```

### Time Window Management

```python
# Custom time window
config = BotSignalLayerConfig(
    name="Short Term Signals",
    time_window_days=1,                # Last 24 hours only
    active_bots_only=True,
    confidence_threshold=0.7           # High confidence only
)
```

### Bot-Specific Filtering

```python
# Filter for specific bots
bot_filter = BotFilterConfig(
    bot_ids=[1, 2, 5],                # Specific bot IDs
    symbols=['BTCUSDT'],
    active_only=True
)

config = BotSignalLayerConfig(
    name="Selected Bots",
    bot_filter=bot_filter,
    include_bot_info=True
)
```

## Examples

### Dashboard Integration Example

```python
# dashboard/callbacks/charts.py
from components.charts.layers import (
    create_bot_signal_layer, 
    create_bot_trade_layer,
    get_active_bot_signals
)

@app.callback(
    Output('chart', 'figure'),
    [Input('symbol-dropdown', 'value'),
     Input('show-bot-signals', 'value')]
)
def update_chart_with_bots(symbol, show_bot_signals):
    fig = create_base_chart(symbol)
    
    if 'bot-signals' in show_bot_signals:
        # Add bot signals
        signal_layer = create_bot_signal_layer(
            symbol=symbol,
            active_only=True,
            confidence_threshold=0.3
        )
        fig = signal_layer.render(fig, market_data, symbol=symbol)
        
        # Add bot trades
        trade_layer = create_bot_trade_layer(
            symbol=symbol,
            active_only=True,
            show_pnl=True
        )
        fig = trade_layer.render(fig, market_data, symbol=symbol)
    
    return fig
```

### Custom Bot Analysis

```python
# Custom analysis for specific strategy
def analyze_momentum_strategy(symbol: str, days_back: int = 30):
    """Analyze momentum strategy performance for a symbol."""
    
    # Get momentum bot signals
    signals = get_bot_signals_by_strategy(
        strategy_name='momentum',
        symbol=symbol,
        days_back=days_back
    )
    
    # Get performance summary
    performance = get_bot_performance_summary(days_back=days_back)
    
    # Create visualizations
    signal_layer = create_bot_signal_layer(
        symbol=symbol,
        active_only=False,  # Include all momentum bots
        time_window_days=days_back
    )
    
    return {
        'signals': signals,
        'performance': performance,
        'layer': signal_layer
    }

# Usage
analysis = analyze_momentum_strategy('BTCUSDT', days_back=14)
```

### Real-time Monitoring Setup

```python
# Real-time bot monitoring dashboard component
def create_realtime_bot_monitor(symbols: List[str]):
    """Create real-time bot monitoring charts."""
    
    charts = {}
    
    for symbol in symbols:
        # Get latest bot data
        active_signals = get_active_bot_signals(
            symbol=symbol,
            days_back=1,  # Last 24 hours
            min_confidence=0.5
        )
        
        # Create monitoring layers
        signal_layer = create_bot_signal_layer(
            symbol=symbol,
            active_only=True,
            time_window_days=1
        )
        
        trade_layer = create_bot_trade_layer(
            symbol=symbol,
            active_only=True,
            show_pnl=True,
            time_window_days=1
        )
        
        charts[symbol] = {
            'signal_layer': signal_layer,
            'trade_layer': trade_layer,
            'active_signals': len(active_signals)
        }
    
    return charts
```

## Best Practices

### Performance Optimization

```python
# 1. Use appropriate time windows
config = BotSignalLayerConfig(
    time_window_days=7,  # Don't fetch more data than needed
    confidence_threshold=0.3  # Filter low-confidence signals
)

# 2. Filter by active bots only when possible
bot_filter = BotFilterConfig(
    active_only=True,    # Reduces database queries
    symbols=['BTCUSDT']  # Specific symbols only
)

# 3. Reuse integration instances
integration = BotSignalLayerIntegration()  # Create once
# Use multiple times for different symbols
```

### Error Handling

```python
try:
    bot_layers = create_complete_bot_layers('BTCUSDT')
    
    if not bot_layers['success']:
        logger.warning(f"Bot layer creation failed: {bot_layers.get('error')}")
        # Fallback to manual signal layer
        signal_layer = TradingSignalLayer()
    else:
        signal_layer = bot_layers['layers']['signals']
        
except Exception as e:
    logger.error(f"Bot integration error: {e}")
    # Graceful degradation
    signal_layer = TradingSignalLayer()
```

### Database Connection Management

```python
# The bot integration handles database connections automatically
# But for custom queries, follow these patterns:

from database.connection import get_session

def custom_bot_query():
    try:
        with get_session() as session:
            # Your database operations
            result = session.query(Bot).filter(...).all()
            return result
    except Exception as e:
        logger.error(f"Database query failed: {e}")
        return []
```

## Troubleshooting

### Common Issues

1. **No signals showing on chart**
   ```python
   # Check if bots exist for symbol
   service = BotDataService()
   bots = service.get_bots(BotFilterConfig(symbols=['BTCUSDT']))
   print(f"Found {len(bots)} bots for BTCUSDT")
   
   # Check signal count
   signals = get_active_bot_signals('BTCUSDT', days_back=7)
   print(f"Found {len(signals)} signals in last 7 days")
   ```

2. **Database connection errors**
   ```python
   # Test database connection
   try:
       from database.connection import get_session
       with get_session() as session:
           print("Database connection successful")
   except Exception as e:
       print(f"Database connection failed: {e}")
   ```

3. **Performance issues with large datasets**
   ```python
   # Reduce time window
   config = BotSignalLayerConfig(
       time_window_days=3,  # Reduced from 7
       confidence_threshold=0.5  # Higher threshold
   )
   
   # Filter by specific strategies
   bot_filter = BotFilterConfig(
       strategies=['momentum'],  # Specific strategy only
       active_only=True
   )
   ```

### Debug Mode

```python
import logging

# Enable debug logging for bot integration
logging.getLogger('bot_integration').setLevel(logging.DEBUG)
logging.getLogger('bot_enhanced_layers').setLevel(logging.DEBUG)

# This will show detailed information about:
# - Database queries
# - Data fetching operations
# - Filter applications
# - Performance metrics
```

### Testing Bot Integration

```python
# Test bot integration components
from tests.test_signal_layers import TestBotIntegration

# Run specific bot integration tests
pytest.main(['-v', 'tests/test_signal_layers.py::TestBotIntegration'])

# Test with mock data
def test_bot_integration():
    config = BotSignalLayerConfig(
        name="Test Bot Signals",
        auto_fetch_data=False  # Use manual data for testing
    )
    
    layer = BotIntegratedSignalLayer(config)
    
    # Provide test data
    test_signals = pd.DataFrame({
        'timestamp': [datetime.now()],
        'signal_type': ['buy'],
        'price': [50000],
        'confidence': [0.8],
        'bot_name': ['Test Bot']
    })
    
    fig = go.Figure()
    result = layer.render(fig, market_data, signals=test_signals)
    
    assert len(result.data) > 0
```

## API Reference

### Core Classes

- **`BotDataService`** - Main service for database operations
- **`BotSignalLayerIntegration`** - Chart-specific integration utilities
- **`BotIntegratedSignalLayer`** - Auto-fetching signal layer
- **`BotIntegratedTradeLayer`** - Auto-fetching trade layer
- **`BotMultiLayerIntegration`** - Multi-bot layer management

### Configuration Classes

- **`BotFilterConfig`** - Bot filtering configuration
- **`BotSignalLayerConfig`** - Signal layer configuration with bot options
- **`BotTradeLayerConfig`** - Trade layer configuration with bot options

### Convenience Functions

- **`create_bot_signal_layer()`** - Quick bot signal layer creation
- **`create_bot_trade_layer()`** - Quick bot trade layer creation
- **`create_complete_bot_layers()`** - Complete bot layer set
- **`get_active_bot_signals()`** - Get signals from active bots
- **`get_active_bot_trades()`** - Get trades from active bots
- **`get_bot_signals_by_strategy()`** - Get signals by strategy
- **`get_bot_performance_summary()`** - Get performance analytics

For complete API documentation, see the module docstrings in:
- `components/charts/layers/bot_integration.py`
- `components/charts/layers/bot_enhanced_layers.py`