TCPDashboard/tasks/4.0-strategy-engine-foundation.md

Relevant Files

• strategies/__init__.py - Strategy package initialization and exports
• strategies/base.py - BaseStrategy abstract class following BaseIndicator pattern
• strategies/factory.py - Strategy factory/registry system for dynamic strategy loading
• strategies/manager.py - StrategyManager class for user-defined strategies (mirrors IndicatorManager)
• strategies/implementations/__init__.py - Strategy implementations package initialization
• strategies/implementations/ema_crossover.py - EMA Crossover strategy implementation
• strategies/implementations/rsi.py - RSI-based momentum strategy implementation
• strategies/implementations/macd.py - MACD trend following strategy implementation
• strategies/utils.py - Strategy utility functions and helpers
• strategies/data_types.py - Strategy-specific data types and signal definitions
• config/strategies/templates/ - Directory for JSON strategy templates
• config/strategies/templates/ema_crossover_template.json - EMA crossover strategy template with schema
• config/strategies/templates/rsi_template.json - RSI strategy template with schema
• config/strategies/templates/macd_template.json - MACD strategy template with schema
• config/strategies/user_strategies/ - Directory for user-defined strategy configurations
• config/strategies/config_utils.py - Strategy configuration utilities and validation
• database/models.py - Updated to include strategy signals table definition
• database/repositories/strategy_repository.py - Strategy signals repository following repository pattern
• database/operations.py - Updated to include strategy operations access
• database/migrations/versions/add_strategy_signals_table.py - Alembic migration for strategy signals table
• components/charts/layers/strategy_signals.py - Strategy signal chart layer for visualization
• components/charts/data_integration.py - Updated to include strategy data integration
• strategies/data_integration.py - Strategy data integration with indicator orchestration and caching
• strategies/validation.py - Strategy signal validation and quality assurance
• strategies/batch_processing.py - Batch processing engine for backtesting multiple strategies across large datasets
• strategies/realtime_execution.py - Real-time strategy execution pipeline for live signal generation
• dashboard/callbacks/realtime_strategies.py - Dashboard callbacks for real-time strategy integration
• tests/strategies/test_base_strategy.py - Unit tests for BaseStrategy abstract class
• tests/strategies/test_strategy_factory.py - Unit tests for strategy factory system
• tests/strategies/test_strategy_manager.py - Unit tests for StrategyManager class
• tests/strategies/implementations/test_ema_crossover.py - Unit tests for EMA Crossover strategy
• tests/strategies/implementations/test_rsi.py - Unit tests for RSI strategy
• tests/strategies/implementations/test_macd.py - Unit tests for MACD strategy
• tests/strategies/test_data_integration.py - Unit tests for strategy data integration
• tests/strategies/test_validation.py - Unit tests for strategy signal validation
• tests/strategies/test_batch_processing.py - Unit tests for batch processing capabilities
• tests/strategies/test_realtime_execution.py - Unit tests for real-time execution pipeline
• tests/database/test_strategy_repository.py - Unit tests for strategy repository

Notes

• Strict Adherence to Indicator Patterns: The strategy engine components (BaseStrategy, StrategyFactory, StrategyManager, Strategy implementations, and configurations) MUST strictly mirror the existing data/common/indicators/ module's structure, factory approach, and configuration management. This ensures consistency and simplifies development.
• Database Segregation for Signals: The newly created strategy_signals table is exclusively for strategy analysis and backtesting results, distinct from the existing signals table which is for live bot trading operations. Maintain this clear separation.
• Initial Full Recalculation: For real-time strategy execution, strategies will initially recalculate completely on each new candle, similar to how technical indicators currently operate. Optimizations for incremental updates can be considered in a later phase.
• Multi-timeframe Support: Strategies should be designed to support and utilize market data from multiple timeframes, following the pattern established by indicators that can consume data from different timeframes.
• Exclusive Use of Repository Pattern: All database interactions, including storing and retrieving strategy signals and run data, must be performed exclusively through the StrategyRepository and other existing repositories. Avoid raw SQL queries.
• JSON-based Configuration: Strategy parameters and configurations are to be managed via JSON files within config/strategies/, aligning with the existing configuration system for indicators and other components.
• Layered Chart Integration: Strategy signals and performance visualizations will be integrated into the dashboard as a new chart layer, utilizing the existing modular chart system.
• Comprehensive Testing: Ensure that all new classes, functions, and modules within the strategy engine have corresponding unit tests placed in the tests/strategies/ directory, following established testing conventions.

Decisions

1. Vectorized vs. Iterative Calculation

• Decision: Refactored strategy signal detection to use vectorized Pandas operations (e.g., shift(), boolean indexing) instead of iterative Python loops.
• Reasoning: Significantly improves performance for signal generation, especially with large datasets, while maintaining identical results as verified by dedicated tests.
• Impact: All core strategy implementations (EMA Crossover, RSI, MACD) now leverage vectorized functions for their primary signal detection logic.

2. Indicator Key Generation Consistency

• Decision: Centralized the _create_indicator_key logic into a shared utility function create_indicator_key() in strategies/utils.py.
• Reasoning: Eliminates code duplication in StrategyFactory and individual strategy implementations, ensuring consistent key generation and easier maintenance if indicator naming conventions change.
• Impact: StrategyFactory and all strategy implementations now use this shared utility for generating unique indicator keys.

3. Removal of calculate_multiple_strategies

• Decision: The calculate_multiple_strategies method was removed from strategies/factory.py.
• Reasoning: This functionality is not immediately required for the current phase of development and can be re-introduced later when needed, to simplify the codebase and testing efforts.
• Impact: The StrategyFactory now focuses on calculating signals for individual strategies, simplifying its interface and reducing initial complexity.

4. strategy_name in Concrete Strategy __init__

• Decision: Updated the __init__ methods of concrete strategy implementations (e.g., EMAStrategy, RSIStrategy, MACDStrategy) to accept and pass strategy_name to BaseStrategy.__init__.
• Reasoning: Ensures consistency with the BaseStrategy abstract class, which now requires strategy_name during initialization, providing a clear identifier for each strategy instance.
• Impact: All strategy implementations now correctly initialize their strategy_name via the base class, standardizing strategy identification across the engine.

5. Database Schema Design for Strategy Analysis

• Decision: Created separate strategy_signals and strategy_runs tables distinct from the existing signals table used for bot operations.
• Reasoning: Maintains clear separation between live bot trading signals and strategy analysis/backtesting data, avoiding conflicts and ensuring data integrity for different use cases.
• Impact: Strategy analysis can be performed independently without affecting live trading operations, with dedicated tables optimized for analytical queries and data retention policies.

6. Repository Pattern Integration

• Decision: Implemented StrategyRepository following the established BaseRepository pattern and integrated it into the centralized DatabaseOperations class.
• Reasoning: Maintains consistency with existing database access patterns, ensures proper session management, and provides a clean API for strategy data operations.
• Impact: All strategy database operations follow the same patterns as other modules, with proper error handling, logging, and transaction management.

7. Vectorized Data Integration

• Decision: Implement vectorized approaches in StrategyDataIntegrator for DataFrame construction, indicator batching, and multi-strategy processing while maintaining iterative interfaces for backward compatibility.
• Reasoning: Significant performance improvements for backtesting and bulk analysis scenarios, better memory efficiency with pandas operations, and preparation for multi-strategy batch processing capabilities.
• Impact: Enhanced performance for large datasets while maintaining existing single-strategy interfaces. Sets foundation for efficient multi-strategy and multi-timeframe processing in future phases.

8. Single-Strategy Orchestration Focus

• Decision: Implement strategy calculation orchestration focused on single-strategy optimization with indicator dependency resolution, avoiding premature multi-strategy complexity.
• Reasoning: Multi-strategy coordination is better handled at the backtesting layer or through parallelization. Single-strategy optimization provides immediate benefits while keeping code maintainable and focused.
• Impact: Cleaner, more maintainable code with optimized single-strategy performance. Provides foundation for future backtester-level parallelization without architectural complexity.

9. Indicator Warm-up Handling for Streaming Batch Processing

• Decision: Implemented dynamic warm-up period calculation and overlapping windows with result trimming for streaming batch processing.
• Reasoning: To ensure accurate indicator calculations and prevent false signals when processing large datasets in chunks, as indicators require a certain amount of historical data to 'warm up'.
• Impact: Guarantees correct backtest results for strategies relying on indicators with warm-up periods, even when using memory-efficient streaming. Automatically adjusts chunk processing to include necessary historical context and removes duplicate/invalid initial signals.

10. Real-time Strategy Execution Architecture

• Decision: Implemented event-driven real-time strategy execution pipeline with signal broadcasting, chart integration, and concurrent processing capabilities.
• Reasoning: Real-time strategy execution requires different architecture than batch processing - event-driven triggers, background signal processing, throttled chart updates, and integration with existing dashboard refresh cycles.
• Impact: Enables live strategy signal generation that integrates seamlessly with the existing chart system. Provides concurrent strategy execution, real-time signal storage, error handling with automatic strategy disabling, and performance monitoring for production use.

11. Exclusion of "HOLD" Signals from All Signal Storage

• Decision: Modified signal storage mechanisms across both real-time and batch processing to exclude "HOLD" signals, only persisting explicit "BUY" or "SELL" signals in the database.
• Reasoning: To significantly reduce data volume and improve storage/query performance across all execution types, as the absence of a BUY/SELL signal implicitly means "HOLD" and can be inferred during analysis or visualization.
• Impact: Leads to more efficient database usage and faster data retrieval for all stored signals. Requires careful handling in visualization and backtesting components to correctly interpret gaps as "HOLD" periods.

Tasks

• 1.0 Core Strategy Foundation Setup

  • 1.1 Create strategies/ directory structure following indicators pattern
  • 1.2 Implement BaseStrategy abstract class in strategies/base.py with calculate() and get_required_indicators() methods
  • 1.3 Create strategies/data_types.py with StrategySignal, SignalType, and StrategyResult classes
  • 1.4 Implement StrategyFactory class in strategies/factory.py for dynamic strategy loading and registration
  • 1.5 Create strategy implementations directory strategies/implementations/
  • 1.6 Implement EMAStrategy in strategies/implementations/ema_crossover.py as reference implementation
  • 1.7 Implement RSIStrategy in strategies/implementations/rsi.py for momentum-based signals
  • 1.8 Implement MACDStrategy in strategies/implementations/macd.py for trend-following signals
  • 1.9 Create strategies/utils.py with helper functions for signal validation and processing
  • 1.10 Create comprehensive unit tests for all strategy foundation components

• 2.0 Strategy Configuration System

  • 2.1 Create config/strategies/ directory structure mirroring indicators configuration
  • 2.2 Implement config/strategies/config_utils.py with configuration validation and loading functions
  • 2.3 Create JSON schema definitions for strategy parameters and validation rules
  • 2.4 Create strategy templates in config/strategies/templates/ for common strategy configurations
  • 2.5 Implement StrategyManager class in strategies/manager.py following IndicatorManager pattern
  • 2.6 Add strategy configuration loading and saving functionality with file-based storage
  • 2.7 Create user strategies directory config/strategies/user_strategies/ for custom configurations
  • 2.8 Implement strategy parameter validation and default value handling
  • 2.9 Add configuration export/import functionality for strategy sharing

• 3.0 Database Schema and Repository Layer

  • 3.1 Create new strategy_signals table migration (separate from existing signals table for bot operations)
  • 3.2 Design strategy_signals table with fields: strategy_name, strategy_config, symbol, timeframe, timestamp, signal_type, price, confidence, signal_metadata, run_id
  • 3.3 Create strategy_runs table to track strategy execution sessions for backtesting and analysis
  • 3.4 Implement StrategyRepository class in database/repositories/strategy_repository.py following repository pattern
  • 3.5 Add strategy repository methods for signal storage, retrieval, and batch operations
  • 3.6 Update database/operations.py to include strategy operations access
  • 3.7 Create database indexes for optimal strategy signal queries (strategy_name, symbol, timeframe, timestamp)
  • 3.8 Add data retention policies for strategy signals (configurable cleanup of old analysis data)
  • 3.9 Implement strategy signal aggregation queries for performance analysis

• 4.0 Strategy Data Integration

  • 4.1 Create StrategyDataIntegrator class in new strategies/data_integration.py module
  • 4.2 Implement data loading interface that leverages existing TechnicalIndicators class for indicator dependencies
  • 4.3 Add multi-timeframe data handling for strategies that require indicators from different timeframes
  • 4.4 Implement strategy calculation orchestration with proper indicator dependency resolution
  • 4.5 Create caching layer for computed indicator results to avoid recalculation across strategies
  • 4.6 Add strategy signal generation and validation pipeline
  • 4.7 Implement batch processing capabilities for backtesting large datasets
  • 4.8 Create real-time strategy execution pipeline that integrates with existing chart data refresh
  • 4.9 Add error handling and recovery mechanisms for strategy calculation failures

• 5.0 Chart Integration and Visualization

  • 5.1 Create StrategySignalLayer class in components/charts/layers/strategy_signals.py
  • 5.2 Implement strategy signal visualization with different markers for entry/exit/hold signals
  • 5.3 Add strategy signal layer configuration following existing chart layer patterns
  • 5.4 Update components/charts/data_integration.py to include strategy data loading for charts
  • 5.5 Create strategy selection controls in dashboard for chart overlay
  • 5.6 Implement real-time strategy signal updates in chart refresh cycle
  • 5.7 Add strategy performance metrics display (win rate, signal accuracy, etc.)
  • 5.8 Create strategy signal filtering and display options (signal types, confidence thresholds)
  • 5.9 Implement strategy comparison visualization for multiple strategies on same chart