TCPDashboard/tasks/4.0-strategy-engine-foundation.md

## Relevant Files

- `strategies/__init__.py` - Strategy package initialization and exports
- `strategies/base.py` - BaseStrategy abstract class following BaseIndicator pattern
- `strategies/factory.py` - Strategy factory/registry system for dynamic strategy loading
- `strategies/manager.py` - StrategyManager class for user-defined strategies (mirrors IndicatorManager)
- `strategies/implementations/__init__.py` - Strategy implementations package initialization
- `strategies/implementations/ema_crossover.py` - EMA Crossover strategy implementation
- `strategies/implementations/rsi.py` - RSI-based momentum strategy implementation
- `strategies/implementations/macd.py` - MACD trend following strategy implementation
- `strategies/utils.py` - Strategy utility functions and helpers
- `strategies/data_types.py` - Strategy-specific data types and signal definitions
- `config/strategies/templates/` - Directory for JSON strategy templates
- `config/strategies/templates/ema_crossover_template.json` - EMA crossover strategy template with schema
- `config/strategies/templates/rsi_template.json` - RSI strategy template with schema
- `config/strategies/templates/macd_template.json` - MACD strategy template with schema
- `config/strategies/user_strategies/` - Directory for user-defined strategy configurations
- `config/strategies/config_utils.py` - Strategy configuration utilities and validation
- `database/models.py` - Updated to include strategy signals table definition
- `database/repositories/strategy_repository.py` - Strategy signals repository following repository pattern
- `database/operations.py` - Updated to include strategy operations access
- `database/migrations/versions/add_strategy_signals_table.py` - Alembic migration for strategy signals table
- `components/charts/layers/strategy_signals.py` - Strategy signal chart layer for visualization
- `components/charts/data_integration.py` - Updated to include strategy data integration
- `tests/strategies/test_base_strategy.py` - Unit tests for BaseStrategy abstract class
- `tests/strategies/test_strategy_factory.py` - Unit tests for strategy factory system
- `tests/strategies/test_strategy_manager.py` - Unit tests for StrategyManager class
- `tests/strategies/implementations/test_ema_crossover.py` - Unit tests for EMA Crossover strategy
- `tests/strategies/implementations/test_rsi.py` - Unit tests for RSI strategy
- `tests/strategies/implementations/test_macd.py` - Unit tests for MACD strategy
- `tests/database/test_strategy_repository.py` - Unit tests for strategy repository

### Notes

- **Strict Adherence to Indicator Patterns**: The strategy engine components (BaseStrategy, StrategyFactory, StrategyManager, Strategy implementations, and configurations) MUST strictly mirror the existing `data/common/indicators/` module's structure, factory approach, and configuration management. This ensures consistency and simplifies development.
- **Database Segregation for Signals**: The newly created `strategy_signals` table is exclusively for strategy analysis and backtesting results, distinct from the existing `signals` table which is for live bot trading operations. Maintain this clear separation.
- **Initial Full Recalculation**: For real-time strategy execution, strategies will initially recalculate completely on each new candle, similar to how technical indicators currently operate. Optimizations for incremental updates can be considered in a later phase.
- **Multi-timeframe Support**: Strategies should be designed to support and utilize market data from multiple timeframes, following the pattern established by indicators that can consume data from different timeframes.
- **Exclusive Use of Repository Pattern**: All database interactions, including storing and retrieving strategy signals and run data, must be performed exclusively through the `StrategyRepository` and other existing repositories. Avoid raw SQL queries.
- **JSON-based Configuration**: Strategy parameters and configurations are to be managed via JSON files within `config/strategies/`, aligning with the existing configuration system for indicators and other components.
- **Layered Chart Integration**: Strategy signals and performance visualizations will be integrated into the dashboard as a new chart layer, utilizing the existing modular chart system.
- **Comprehensive Testing**: Ensure that all new classes, functions, and modules within the strategy engine have corresponding unit tests placed in the `tests/strategies/` directory, following established testing conventions.

## Decisions

### 1. Vectorized vs. Iterative Calculation
- **Decision**: Refactored strategy signal detection to use vectorized Pandas operations (e.g., `shift()`, boolean indexing) instead of iterative Python loops.
- **Reasoning**: Significantly improves performance for signal generation, especially with large datasets, while maintaining identical results as verified by dedicated tests.
- **Impact**: All core strategy implementations (EMA Crossover, RSI, MACD) now leverage vectorized functions for their primary signal detection logic.

### 2. Indicator Key Generation Consistency
- **Decision**: Centralized the `_create_indicator_key` logic into a shared utility function `create_indicator_key()` in `strategies/utils.py`.
- **Reasoning**: Eliminates code duplication in `StrategyFactory` and individual strategy implementations, ensuring consistent key generation and easier maintenance if indicator naming conventions change.
- **Impact**: `StrategyFactory` and all strategy implementations now use this shared utility for generating unique indicator keys.

### 3. Removal of `calculate_multiple_strategies`
- **Decision**: The `calculate_multiple_strategies` method was removed from `strategies/factory.py`.
- **Reasoning**: This functionality is not immediately required for the current phase of development and can be re-introduced later when needed, to simplify the codebase and testing efforts.
- **Impact**: The `StrategyFactory` now focuses on calculating signals for individual strategies, simplifying its interface and reducing initial complexity.

### 4. `strategy_name` in Concrete Strategy `__init__`
- **Decision**: Updated the `__init__` methods of concrete strategy implementations (e.g., `EMAStrategy`, `RSIStrategy`, `MACDStrategy`) to accept and pass `strategy_name` to `BaseStrategy.__init__`.
- **Reasoning**: Ensures consistency with the `BaseStrategy` abstract class, which now requires `strategy_name` during initialization, providing a clear identifier for each strategy instance.
- **Impact**: All strategy implementations now correctly initialize their `strategy_name` via the base class, standardizing strategy identification across the engine.

## Tasks

- [x] 1.0 Core Strategy Foundation Setup
  - [x] 1.1 Create `strategies/` directory structure following indicators pattern
  - [x] 1.2 Implement `BaseStrategy` abstract class in `strategies/base.py` with `calculate()` and `get_required_indicators()` methods
  - [x] 1.3 Create `strategies/data_types.py` with `StrategySignal`, `SignalType`, and `StrategyResult` classes
  - [x] 1.4 Implement `StrategyFactory` class in `strategies/factory.py` for dynamic strategy loading and registration
  - [x] 1.5 Create strategy implementations directory `strategies/implementations/`
  - [x] 1.6 Implement `EMAStrategy` in `strategies/implementations/ema_crossover.py` as reference implementation
  - [x] 1.7 Implement `RSIStrategy` in `strategies/implementations/rsi.py` for momentum-based signals
  - [x] 1.8 Implement `MACDStrategy` in `strategies/implementations/macd.py` for trend-following signals
  - [x] 1.9 Create `strategies/utils.py` with helper functions for signal validation and processing
  - [x] 1.10 Create comprehensive unit tests for all strategy foundation components

- [x] 2.0 Strategy Configuration System
  - [x] 2.1 Create `config/strategies/` directory structure mirroring indicators configuration
  - [x] 2.2 Implement `config/strategies/config_utils.py` with configuration validation and loading functions
  - [x] 2.3 Create JSON schema definitions for strategy parameters and validation rules
  - [x] 2.4 Create strategy templates in `config/strategies/templates/` for common strategy configurations
  - [x] 2.5 Implement `StrategyManager` class in `strategies/manager.py` following `IndicatorManager` pattern
  - [x] 2.6 Add strategy configuration loading and saving functionality with file-based storage
  - [x] 2.7 Create user strategies directory `config/strategies/user_strategies/` for custom configurations
  - [x] 2.8 Implement strategy parameter validation and default value handling
  - [x] 2.9 Add configuration export/import functionality for strategy sharing

- [ ] 3.0 Database Schema and Repository Layer
  - [ ] 3.1 Create new `strategy_signals` table migration (separate from existing `signals` table for bot operations)
  - [ ] 3.2 Design `strategy_signals` table with fields: strategy_name, strategy_config, symbol, timeframe, timestamp, signal_type, price, confidence, metadata, run_id
  - [ ] 3.3 Create `strategy_runs` table to track strategy execution sessions for backtesting and analysis
  - [ ] 3.4 Implement `StrategyRepository` class in `database/repositories/strategy_repository.py` following repository pattern
  - [ ] 3.5 Add strategy repository methods for signal storage, retrieval, and batch operations
  - [ ] 3.6 Update `database/operations.py` to include strategy operations access
  - [ ] 3.7 Create database indexes for optimal strategy signal queries (strategy_name, symbol, timeframe, timestamp)
  - [ ] 3.8 Add data retention policies for strategy signals (configurable cleanup of old analysis data)
  - [ ] 3.9 Implement strategy signal aggregation queries for performance analysis

- [ ] 4.0 Strategy Data Integration
  - [ ] 4.1 Create `StrategyDataIntegrator` class in new `strategies/data_integration.py` module
  - [ ] 4.2 Implement data loading interface that leverages existing `TechnicalIndicators` class for indicator dependencies
  - [x] 4.3 Add multi-timeframe data handling for strategies that require indicators from different timeframes
  - [ ] 4.4 Implement strategy calculation orchestration with proper indicator dependency resolution
  - [ ] 4.5 Create caching layer for computed indicator results to avoid recalculation across strategies
  - [ ] 4.6 Add strategy signal generation and validation pipeline
  - [ ] 4.7 Implement batch processing capabilities for backtesting large datasets
  - [ ] 4.8 Create real-time strategy execution pipeline that integrates with existing chart data refresh
  - [ ] 4.9 Add error handling and recovery mechanisms for strategy calculation failures

- [ ] 5.0 Chart Integration and Visualization
  - [ ] 5.1 Create `StrategySignalLayer` class in `components/charts/layers/strategy_signals.py`
  - [ ] 5.2 Implement strategy signal visualization with different markers for entry/exit/hold signals
  - [ ] 5.3 Add strategy signal layer configuration following existing chart layer patterns
  - [ ] 5.4 Update `components/charts/data_integration.py` to include strategy data loading for charts
  - [ ] 5.5 Create strategy selection controls in dashboard for chart overlay
  - [ ] 5.6 Implement real-time strategy signal updates in chart refresh cycle
  - [ ] 5.7 Add strategy performance metrics display (win rate, signal accuracy, etc.)
  - [ ] 5.8 Create strategy signal filtering and display options (signal types, confidence thresholds)
  - [ ] 5.9 Implement strategy comparison visualization for multiple strategies on same chart