TCPDashboard/docs/modules/strategies.md

# Module: Strategies

## Purpose
This module encompasses the core strategy engine for the TCP Trading Platform. It provides a flexible and extensible framework for defining, managing, and executing trading strategies. It includes components for real-time signal generation, batch backtesting, and data integration with market data and technical indicators.

## Public Interface

### Classes
- `strategies.base.BaseStrategy`: Abstract base class for all trading strategies, defining the common interface for `calculate()` and `get_required_indicators()`.
- `strategies.factory.StrategyFactory`: Manages dynamic loading and registration of strategy implementations based on configuration.
- `strategies.manager.StrategyManager`: Handles user-defined strategy configurations, enabling loading, saving, and validation of strategy parameters.
- `strategies.data_integration.StrategyDataIntegrator`: Orchestrates the integration of market data and pre-calculated technical indicators for strategy execution. It handles data fetching, caching, and dependency resolution.
- `strategies.batch_processing.BacktestingBatchProcessor`: Provides capabilities for efficient batch execution of multiple strategies across large historical datasets, including memory management and performance optimization.
- `strategies.realtime_execution.RealTimeStrategyProcessor`: Manages real-time strategy execution, generating signals incrementally as new market data becomes available. It integrates with the dashboard's chart refresh cycle and handles signal broadcasting.
- `strategies.realtime_execution.StrategySignalBroadcaster`: A component of the real-time execution pipeline responsible for distributing generated signals, storing them in the database, and triggering chart updates.

### Functions
- `strategies.realtime_execution.get_realtime_strategy_processor()`: Returns the singleton instance of the `RealTimeStrategyProcessor`.
- `strategies.realtime_execution.initialize_realtime_strategy_system()`: Initializes and starts the real-time strategy execution system.
- `strategies.realtime_execution.shutdown_realtime_strategy_system()`: Shuts down the real-time strategy execution system.

## Usage Examples

### 1. Registering and Executing a Real-time Strategy
```python
from strategies.realtime_execution import initialize_realtime_strategy_system
from config.strategies.config_utils import StrategyConfigurationManager

# Initialize the real-time system (usually done once at application startup)
processor = initialize_realtime_strategy_system()

# Load a strategy configuration (e.g., from a user-defined config or template)
config_manager = StrategyConfigurationManager()
strategy_name = "ema_crossover"
strategy_config = config_manager.load_user_strategy_config(strategy_name) or \
                  config_manager.load_strategy_template(strategy_name)

if strategy_config:
    # Register the strategy for real-time execution on a specific symbol and timeframe
    context_id = processor.register_strategy(
        strategy_name=strategy_name,
        strategy_config=strategy_config,
        symbol="BTC-USDT",
        timeframe="1h"
    )
    print(f"Strategy '{strategy_name}' registered for real-time updates with ID: {context_id}")

    # In a real application, new data would arrive (e.g., via a WebSocket or API poll)
    # and `processor.execute_realtime_update()` would be called from a data refresh callback.
    # Example of manual trigger (for illustration):
    # signals = processor.execute_realtime_update(symbol="BTC-USDT", timeframe="1h")
    # print(f"Generated {len(signals)} signals for {strategy_name}")

# To stop the system (usually done at application shutdown)
# shutdown_realtime_strategy_system()
```

### 2. Running a Backtesting Batch Process
```python
from strategies.batch_processing import BacktestingBatchProcessor, BatchProcessingConfig
from config.strategies.config_utils import StrategyConfigurationManager
import pandas as pd
from datetime import datetime, timedelta, timezone

# Example historical data (in a real scenario, this would come from a database)
# This is a placeholder for demonstration purposes
# Ensure you have actual OHLCVCandle data or a DataFrame that mimics it
example_data = [
    # ... populate with enough OHLCVCandle objects for indicator warm-up
    # For instance, 100 OHLCVCandle objects
]

# Create a mock DataFrame for demonstration, ensure it has 'open', 'high', 'low', 'close', 'volume', 'timestamp'
# For testing, you'd load actual data or generate a synthetic one.
ohlcv_data = []
start_ts = datetime.now(timezone.utc) - timedelta(days=30)
for i in range(300):
    ohlcv_data.append({
        "timestamp": (start_ts + timedelta(hours=i)).isoformat(),
        "open": 100.0 + i * 0.1,
        "high": 101.0 + i * 0.1,
        "low": 99.0 + i * 0.1,
        "close": 100.5 + i * 0.1,
        "volume": 1000.0 + i * 10
    })
market_df = pd.DataFrame(ohlcv_data).set_index("timestamp")
market_df.index = pd.to_datetime(market_df.index).to_pydatetime() # Convert to datetime objects if needed

# Initialize batch processor config
batch_config = BatchProcessingConfig(
    chunk_size=100, # Process in chunks for memory efficiency
    max_concurrent_tasks=2
)
batch_processor = BacktestingBatchProcessor(batch_config)

# Load strategies to backtest
config_manager = StrategyConfigurationManager()
strategies_to_backtest = {
    "ema_crossover": config_manager.load_strategy_template("ema_crossover"),
    "macd": config_manager.load_strategy_template("macd")
}

# Run batch processing
if all(strategies_to_backtest.values()): # Ensure all configs are loaded
    print("Starting batch backtesting...")
    results = batch_processor.process_strategies_batch(
        market_df=market_df,
        strategies_config=strategies_to_backtest,
        symbols=["BTC-USDT", "ETH-USDT"],
        timeframe="1h"
    )
    print("Batch backtesting complete. Results:")
    for strategy_name, symbol_results in results.items():
        for symbol, strategy_results_list in symbol_results.items():
            print(f"  Strategy: {strategy_name}, Symbol: {symbol}, Signals: {len(strategy_results_list)}")
else:
    print("Failed to load all strategy configurations. Skipping batch processing.")
```

## Dependencies

### Internal Dependencies
- `database.operations`: For database interactions (storing signals).
- `data.common.data_types`: Defines common data structures like `OHLCVCandle`.
- `data.common.indicators.TechnicalIndicators`: Provides access to pre-calculated technical indicators.
- `components.charts.data_integration.MarketDataIntegrator`: Used by `StrategyDataIntegrator` to fetch raw market data.
- `utils.logger`: For logging within the module.
- `config.strategies.config_utils`: For loading and managing strategy configurations.

### External Dependencies
- `pandas`: For data manipulation and vectorized operations.
- `datetime`: For handling timestamps and time-related calculations.
- `typing`: For type hints.
- `dataclasses`: For defining data classes like `RealTimeConfig`, `StrategyExecutionContext`, etc.
- `threading`, `queue`, `concurrent.futures`: For managing concurrency and background processing in real-time execution.

## Testing

To run tests for the `strategies` module, navigate to the project root and execute:

```bash
python -m pytest tests/strategies/test_realtime_execution.py -v
python -m pytest tests/strategies/test_batch_processing.py -v
# Add other relevant strategy tests here, e.g., for data_integration, manager, factory
```

## Known Issues

- **Incremental Calculation Optimization**: The `_calculate_incremental_signals` method in `RealTimeStrategyProcessor` currently falls back to full recalculation. True incremental calculation (only processing new candles and updating existing indicator series) is a future optimization.
- **Chart Layer Updates**: While the `StrategySignalBroadcaster` triggers a chart update callback, the actual chart layer integration (`components/charts/layers/strategy_signals.py`) and dynamic signal display on the chart are part of Task 5.0 and not fully implemented within this module.