## Relevant Files - `strategies/__init__.py` - Strategy package initialization and exports - `strategies/base.py` - BaseStrategy abstract class following BaseIndicator pattern - `strategies/factory.py` - Strategy factory/registry system for dynamic strategy loading - `strategies/manager.py` - StrategyManager class for user-defined strategies (mirrors IndicatorManager) - `strategies/implementations/__init__.py` - Strategy implementations package initialization - `strategies/implementations/ema_crossover.py` - EMA Crossover strategy implementation - `strategies/implementations/rsi.py` - RSI-based momentum strategy implementation - `strategies/implementations/macd.py` - MACD trend following strategy implementation - `strategies/utils.py` - Strategy utility functions and helpers - `strategies/data_types.py` - Strategy-specific data types and signal definitions - `config/strategies/templates/` - Directory for JSON strategy templates - `config/strategies/templates/ema_crossover_template.json` - EMA crossover strategy template with schema - `config/strategies/templates/rsi_template.json` - RSI strategy template with schema - `config/strategies/templates/macd_template.json` - MACD strategy template with schema - `config/strategies/user_strategies/` - Directory for user-defined strategy configurations - `config/strategies/config_utils.py` - Strategy configuration utilities and validation - `database/models.py` - Updated to include strategy signals table definition - `database/repositories/strategy_repository.py` - Strategy signals repository following repository pattern - `database/operations.py` - Updated to include strategy operations access - `database/migrations/versions/add_strategy_signals_table.py` - Alembic migration for strategy signals table - `components/charts/layers/strategy_signals.py` - Strategy signal chart layer for visualization - `components/charts/data_integration.py` - Updated to include strategy data integration - `strategies/data_integration.py` - Strategy data integration with indicator orchestration and caching - `strategies/validation.py` - Strategy signal validation and quality assurance - `strategies/batch_processing.py` - Batch processing engine for backtesting multiple strategies across large datasets - `strategies/realtime_execution.py` - Real-time strategy execution pipeline for live signal generation - `dashboard/callbacks/realtime_strategies.py` - Dashboard callbacks for real-time strategy integration - `tests/strategies/test_base_strategy.py` - Unit tests for BaseStrategy abstract class - `tests/strategies/test_strategy_factory.py` - Unit tests for strategy factory system - `tests/strategies/test_strategy_manager.py` - Unit tests for StrategyManager class - `tests/strategies/implementations/test_ema_crossover.py` - Unit tests for EMA Crossover strategy - `tests/strategies/implementations/test_rsi.py` - Unit tests for RSI strategy - `tests/strategies/implementations/test_macd.py` - Unit tests for MACD strategy - `tests/strategies/test_data_integration.py` - Unit tests for strategy data integration - `tests/strategies/test_validation.py` - Unit tests for strategy signal validation - `tests/strategies/test_batch_processing.py` - Unit tests for batch processing capabilities - `tests/strategies/test_realtime_execution.py` - Unit tests for real-time execution pipeline - `tests/database/test_strategy_repository.py` - Unit tests for strategy repository ### Notes - **Strict Adherence to Indicator Patterns**: The strategy engine components (BaseStrategy, StrategyFactory, StrategyManager, Strategy implementations, and configurations) MUST strictly mirror the existing `data/common/indicators/` module's structure, factory approach, and configuration management. This ensures consistency and simplifies development. - **Database Segregation for Signals**: The newly created `strategy_signals` table is exclusively for strategy analysis and backtesting results, distinct from the existing `signals` table which is for live bot trading operations. Maintain this clear separation. - **Initial Full Recalculation**: For real-time strategy execution, strategies will initially recalculate completely on each new candle, similar to how technical indicators currently operate. Optimizations for incremental updates can be considered in a later phase. - **Multi-timeframe Support**: Strategies should be designed to support and utilize market data from multiple timeframes, following the pattern established by indicators that can consume data from different timeframes. - **Exclusive Use of Repository Pattern**: All database interactions, including storing and retrieving strategy signals and run data, must be performed exclusively through the `StrategyRepository` and other existing repositories. Avoid raw SQL queries. - **JSON-based Configuration**: Strategy parameters and configurations are to be managed via JSON files within `config/strategies/`, aligning with the existing configuration system for indicators and other components. - **Layered Chart Integration**: Strategy signals and performance visualizations will be integrated into the dashboard as a new chart layer, utilizing the existing modular chart system. - **Comprehensive Testing**: Ensure that all new classes, functions, and modules within the strategy engine have corresponding unit tests placed in the `tests/strategies/` directory, following established testing conventions. ## Decisions ### 1. Vectorized vs. Iterative Calculation - **Decision**: Refactored strategy signal detection to use vectorized Pandas operations (e.g., `shift()`, boolean indexing) instead of iterative Python loops. - **Reasoning**: Significantly improves performance for signal generation, especially with large datasets, while maintaining identical results as verified by dedicated tests. - **Impact**: All core strategy implementations (EMA Crossover, RSI, MACD) now leverage vectorized functions for their primary signal detection logic. ### 2. Indicator Key Generation Consistency - **Decision**: Centralized the `_create_indicator_key` logic into a shared utility function `create_indicator_key()` in `strategies/utils.py`. - **Reasoning**: Eliminates code duplication in `StrategyFactory` and individual strategy implementations, ensuring consistent key generation and easier maintenance if indicator naming conventions change. - **Impact**: `StrategyFactory` and all strategy implementations now use this shared utility for generating unique indicator keys. ### 3. Removal of `calculate_multiple_strategies` - **Decision**: The `calculate_multiple_strategies` method was removed from `strategies/factory.py`. - **Reasoning**: This functionality is not immediately required for the current phase of development and can be re-introduced later when needed, to simplify the codebase and testing efforts. - **Impact**: The `StrategyFactory` now focuses on calculating signals for individual strategies, simplifying its interface and reducing initial complexity. ### 4. `strategy_name` in Concrete Strategy `__init__` - **Decision**: Updated the `__init__` methods of concrete strategy implementations (e.g., `EMAStrategy`, `RSIStrategy`, `MACDStrategy`) to accept and pass `strategy_name` to `BaseStrategy.__init__`. - **Reasoning**: Ensures consistency with the `BaseStrategy` abstract class, which now requires `strategy_name` during initialization, providing a clear identifier for each strategy instance. - **Impact**: All strategy implementations now correctly initialize their `strategy_name` via the base class, standardizing strategy identification across the engine. ### 5. Database Schema Design for Strategy Analysis - **Decision**: Created separate `strategy_signals` and `strategy_runs` tables distinct from the existing `signals` table used for bot operations. - **Reasoning**: Maintains clear separation between live bot trading signals and strategy analysis/backtesting data, avoiding conflicts and ensuring data integrity for different use cases. - **Impact**: Strategy analysis can be performed independently without affecting live trading operations, with dedicated tables optimized for analytical queries and data retention policies. ### 6. Repository Pattern Integration - **Decision**: Implemented `StrategyRepository` following the established `BaseRepository` pattern and integrated it into the centralized `DatabaseOperations` class. - **Reasoning**: Maintains consistency with existing database access patterns, ensures proper session management, and provides a clean API for strategy data operations. - **Impact**: All strategy database operations follow the same patterns as other modules, with proper error handling, logging, and transaction management. ### 7. Vectorized Data Integration - **Decision**: Implement vectorized approaches in `StrategyDataIntegrator` for DataFrame construction, indicator batching, and multi-strategy processing while maintaining iterative interfaces for backward compatibility. - **Reasoning**: Significant performance improvements for backtesting and bulk analysis scenarios, better memory efficiency with pandas operations, and preparation for multi-strategy batch processing capabilities. - **Impact**: Enhanced performance for large datasets while maintaining existing single-strategy interfaces. Sets foundation for efficient multi-strategy and multi-timeframe processing in future phases. ### 8. Single-Strategy Orchestration Focus - **Decision**: Implement strategy calculation orchestration focused on single-strategy optimization with indicator dependency resolution, avoiding premature multi-strategy complexity. - **Reasoning**: Multi-strategy coordination is better handled at the backtesting layer or through parallelization. Single-strategy optimization provides immediate benefits while keeping code maintainable and focused. - **Impact**: Cleaner, more maintainable code with optimized single-strategy performance. Provides foundation for future backtester-level parallelization without architectural complexity. ### 9. Indicator Warm-up Handling for Streaming Batch Processing - **Decision**: Implemented dynamic warm-up period calculation and overlapping windows with result trimming for streaming batch processing. - **Reasoning**: To ensure accurate indicator calculations and prevent false signals when processing large datasets in chunks, as indicators require a certain amount of historical data to 'warm up'. - **Impact**: Guarantees correct backtest results for strategies relying on indicators with warm-up periods, even when using memory-efficient streaming. Automatically adjusts chunk processing to include necessary historical context and removes duplicate/invalid initial signals. ### 10. Real-time Strategy Execution Architecture - **Decision**: Implemented event-driven real-time strategy execution pipeline with signal broadcasting, chart integration, and concurrent processing capabilities. - **Reasoning**: Real-time strategy execution requires different architecture than batch processing - event-driven triggers, background signal processing, throttled chart updates, and integration with existing dashboard refresh cycles. - **Impact**: Enables live strategy signal generation that integrates seamlessly with the existing chart system. Provides concurrent strategy execution, real-time signal storage, error handling with automatic strategy disabling, and performance monitoring for production use. ### 11. Exclusion of "HOLD" Signals from All Signal Storage - **Decision**: Modified signal storage mechanisms across both real-time and batch processing to exclude "HOLD" signals, only persisting explicit "BUY" or "SELL" signals in the database. - **Reasoning**: To significantly reduce data volume and improve storage/query performance across all execution types, as the absence of a BUY/SELL signal implicitly means "HOLD" and can be inferred during analysis or visualization. - **Impact**: Leads to more efficient database usage and faster data retrieval for all stored signals. Requires careful handling in visualization and backtesting components to correctly interpret gaps as "HOLD" periods. ## Tasks - [x] 1.0 Core Strategy Foundation Setup - [x] 1.1 Create `strategies/` directory structure following indicators pattern - [x] 1.2 Implement `BaseStrategy` abstract class in `strategies/base.py` with `calculate()` and `get_required_indicators()` methods - [x] 1.3 Create `strategies/data_types.py` with `StrategySignal`, `SignalType`, and `StrategyResult` classes - [x] 1.4 Implement `StrategyFactory` class in `strategies/factory.py` for dynamic strategy loading and registration - [x] 1.5 Create strategy implementations directory `strategies/implementations/` - [x] 1.6 Implement `EMAStrategy` in `strategies/implementations/ema_crossover.py` as reference implementation - [x] 1.7 Implement `RSIStrategy` in `strategies/implementations/rsi.py` for momentum-based signals - [x] 1.8 Implement `MACDStrategy` in `strategies/implementations/macd.py` for trend-following signals - [x] 1.9 Create `strategies/utils.py` with helper functions for signal validation and processing - [x] 1.10 Create comprehensive unit tests for all strategy foundation components - [x] 2.0 Strategy Configuration System - [x] 2.1 Create `config/strategies/` directory structure mirroring indicators configuration - [x] 2.2 Implement `config/strategies/config_utils.py` with configuration validation and loading functions - [x] 2.3 Create JSON schema definitions for strategy parameters and validation rules - [x] 2.4 Create strategy templates in `config/strategies/templates/` for common strategy configurations - [x] 2.5 Implement `StrategyManager` class in `strategies/manager.py` following `IndicatorManager` pattern - [x] 2.6 Add strategy configuration loading and saving functionality with file-based storage - [x] 2.7 Create user strategies directory `config/strategies/user_strategies/` for custom configurations - [x] 2.8 Implement strategy parameter validation and default value handling - [x] 2.9 Add configuration export/import functionality for strategy sharing - [x] 3.0 Database Schema and Repository Layer - [x] 3.1 Create new `strategy_signals` table migration (separate from existing `signals` table for bot operations) - [x] 3.2 Design `strategy_signals` table with fields: strategy_name, strategy_config, symbol, timeframe, timestamp, signal_type, price, confidence, signal_metadata, run_id - [x] 3.3 Create `strategy_runs` table to track strategy execution sessions for backtesting and analysis - [x] 3.4 Implement `StrategyRepository` class in `database/repositories/strategy_repository.py` following repository pattern - [x] 3.5 Add strategy repository methods for signal storage, retrieval, and batch operations - [x] 3.6 Update `database/operations.py` to include strategy operations access - [x] 3.7 Create database indexes for optimal strategy signal queries (strategy_name, symbol, timeframe, timestamp) - [x] 3.8 Add data retention policies for strategy signals (configurable cleanup of old analysis data) - [x] 3.9 Implement strategy signal aggregation queries for performance analysis - [x] 4.0 Strategy Data Integration - [x] 4.1 Create `StrategyDataIntegrator` class in new `strategies/data_integration.py` module - [x] 4.2 Implement data loading interface that leverages existing `TechnicalIndicators` class for indicator dependencies - [x] 4.3 Add multi-timeframe data handling for strategies that require indicators from different timeframes - [x] 4.4 Implement strategy calculation orchestration with proper indicator dependency resolution - [x] 4.5 Create caching layer for computed indicator results to avoid recalculation across strategies - [x] 4.6 Add strategy signal generation and validation pipeline - [x] 4.7 Implement batch processing capabilities for backtesting large datasets - [x] 4.8 Create real-time strategy execution pipeline that integrates with existing chart data refresh - [x] 4.9 Add error handling and recovery mechanisms for strategy calculation failures - [ ] 5.0 Chart Integration and Visualization - [ ] 5.1 Create `StrategySignalLayer` class in `components/charts/layers/strategy_signals.py` - [ ] 5.2 Implement strategy signal visualization with different markers for entry/exit/hold signals - [ ] 5.3 Add strategy signal layer configuration following existing chart layer patterns - [ ] 5.4 Update `components/charts/data_integration.py` to include strategy data loading for charts - [ ] 5.5 Create strategy selection controls in dashboard for chart overlay - [ ] 5.6 Implement real-time strategy signal updates in chart refresh cycle - [ ] 5.7 Add strategy performance metrics display (win rate, signal accuracy, etc.) - [ ] 5.8 Create strategy signal filtering and display options (signal types, confidence thresholds) - [ ] 5.9 Implement strategy comparison visualization for multiple strategies on same chart