Cycles/test/backtest/README.md

Strategy Backtest Runner

A comprehensive and efficient backtest runner for executing predefined trading strategies with advanced visualization and analysis capabilities.

Overview

The Strategy Backtest Runner (strategy_run.py) executes specific trading strategies with predefined parameters defined in a JSON configuration file. Unlike the parameter optimization script, this runner focuses on testing and comparing specific strategy configurations with detailed market analysis and visualization.

Features

• JSON Configuration: Define strategies and parameters in easy-to-edit JSON files
• Multiple Strategy Support: Run multiple strategies in sequence with a single command
• All Strategy Types: Support for MetaTrend, BBRS, and Random strategies
• Organized Results: Automatic folder structure creation for each run
• Advanced Visualization: Detailed plots showing portfolio performance and market context
• Full Market Data Integration: Continuous price charts with buy/sell signals overlay
• Signal Export: Complete buy/sell signal data exported to CSV files
• Real-time File Saving: Individual strategy results saved immediately upon completion
• Comprehensive Analysis: Multiple plot types for thorough performance analysis
• Detailed Results: Comprehensive result reporting with CSV and JSON export
• Result Analysis: Automatic summary generation and performance comparison
• Error Handling: Robust error handling with detailed logging
• Flexible Configuration: Support for different data files, date ranges, and trader parameters

Usage

Basic Usage

# Run strategies from a configuration file
python test/backtest/strategy_run.py --config configs/strategy/example_strategies.json

# Save results to a custom directory
python test/backtest/strategy_run.py --config configs/strategy/my_strategies.json --results-dir my_results

# Enable verbose logging
python test/backtest/strategy_run.py --config configs/strategy/example_strategies.json --verbose

Enhanced Analysis Features

Each run automatically generates:

• Organized folder structure with timestamp for easy management
• Real-time file saving - results saved immediately after each strategy completes
• Full market data visualization - continuous price charts show complete market context
• Signal tracking - all buy/sell decisions exported with precise timing and pricing
• Multi-layered analysis - from individual trade details to portfolio-wide comparisons
• Professional plots - high-resolution (300 DPI) charts suitable for reports and presentations

Create Example Configuration

# Create an example configuration file
python test/backtest/strategy_run.py --create-example configs/example_strategies.json

Configuration File Format

The configuration file uses JSON format with two main sections:

Backtest Settings

{
  "backtest_settings": {
    "data_file": "btcusd_1-min_data.csv",
    "data_dir": "data",
    "start_date": "2023-01-01",
    "end_date": "2023-01-31",
    "initial_usd": 10000
  }
}

Strategy Definitions

{
  "strategies": [
    {
      "name": "MetaTrend_Conservative",
      "type": "metatrend",
      "params": {
        "supertrend_periods": [12, 10, 11],
        "supertrend_multipliers": [3.0, 1.0, 2.0],
        "min_trend_agreement": 0.8,
        "timeframe": "15min"
      },
      "trader_params": {
        "stop_loss_pct": 0.02,
        "portfolio_percent_per_trade": 0.5
      }
    }
  ]
}

Strategy Types

MetaTrend Strategy

Parameters:

• supertrend_periods: List of periods for multiple supertrend indicators
• supertrend_multipliers: List of multipliers for supertrend indicators
• min_trend_agreement: Minimum agreement threshold between indicators (0.0-1.0)
• timeframe: Data aggregation timeframe ("1min", "5min", "15min", "30min", "1h")

BBRS Strategy

Parameters:

• bb_length: Bollinger Bands period
• bb_std: Bollinger Bands standard deviation multiplier
• rsi_length: RSI period
• rsi_overbought: RSI overbought threshold
• rsi_oversold: RSI oversold threshold
• timeframe: Data aggregation timeframe

Random Strategy

Parameters:

• signal_probability: Probability of generating a signal (0.0-1.0)
• timeframe: Data aggregation timeframe

Trader Parameters

All strategies support these trader parameters:

• stop_loss_pct: Stop loss percentage (e.g., 0.02 for 2%)
• portfolio_percent_per_trade: Percentage of portfolio to use per trade (0.0-1.0)

Results Organization

Each run creates an organized folder structure for easy navigation and analysis:

results/
└── [config_name]_[timestamp]/
    ├── strategy_1_[strategy_name].json                    # Individual strategy data
    ├── strategy_1_[strategy_name]_plot.png               # 4-panel performance plot
    ├── strategy_1_[strategy_name]_detailed_plot.png      # 3-panel market analysis
    ├── strategy_1_[strategy_name]_trades.csv             # Trade details
    ├── strategy_1_[strategy_name]_signals.csv            # All buy/sell signals
    ├── strategy_2_[strategy_name].*                      # Second strategy files
    ├── ...                                               # Additional strategies
    ├── summary.csv                                       # Strategy comparison table
    ├── summary_plot.png                                  # Multi-strategy comparison
    └── summary_*.json                                    # Comprehensive results

Visualization Types

The runner generates three types of plots for comprehensive analysis:

1. Individual Strategy Plot (4-Panel)

• Equity Curve: Portfolio value over time
• Trade P&L: Individual trade profits/losses
• Drawdown: Portfolio drawdown visualization
• Statistics: Strategy performance summary

2. Detailed Market Analysis Plot (3-Panel)

• Portfolio Signals: Portfolio value with buy/sell signal markers
• Market Price: Full continuous market price with entry/exit points
• Combined View: Dual-axis plot showing market vs portfolio performance

3. Summary Comparison Plot (4-Panel)

• Returns Comparison: Total returns across all strategies
• Trade Counts: Number of trades per strategy
• Risk vs Return: Win rate vs maximum drawdown scatter plot
• Statistics Table: Comprehensive performance metrics

Output Files

The runner generates comprehensive output files organized in dedicated folders:

Individual Strategy Files (per strategy)

• strategy_N_[name].json: Complete strategy data and metadata
• strategy_N_[name]_plot.png: 4-panel performance analysis plot
• strategy_N_[name]_detailed_plot.png: 3-panel market context plot
• strategy_N_[name]_trades.csv: Detailed trade information
• strategy_N_[name]_signals.csv: All buy/sell signals with timestamps

Summary Files (per run)

• summary.csv: Strategy comparison table
• summary_plot.png: Multi-strategy comparison visualization
• summary_*.json: Comprehensive results and metadata

Signal Data Format

Each signal CSV contains:

• signal_id: Unique signal identifier
• signal_type: BUY or SELL
• time: Signal timestamp
• price: Execution price
• trade_id: Associated trade number
• quantity: Trade quantity
• value: Trade value (quantity × price)
• strategy: Strategy name

Example Configurations

Simple MetaTrend Test

{
  "backtest_settings": {
    "data_file": "btcusd_1-min_data.csv",
    "start_date": "2023-01-01",
    "end_date": "2023-01-07",
    "initial_usd": 10000
  },
  "strategies": [
    {
      "name": "MetaTrend_Test",
      "type": "metatrend",
      "params": {
        "supertrend_periods": [12, 10],
        "supertrend_multipliers": [3.0, 1.0],
        "min_trend_agreement": 0.5,
        "timeframe": "15min"
      },
      "trader_params": {
        "stop_loss_pct": 0.02,
        "portfolio_percent_per_trade": 0.5
      }
    }
  ]
}

Multiple Strategy Comparison

{
  "backtest_settings": {
    "data_file": "btcusd_1-min_data.csv",
    "start_date": "2023-01-01",
    "end_date": "2023-01-31",
    "initial_usd": 10000
  },
  "strategies": [
    {
      "name": "Conservative_MetaTrend",
      "type": "metatrend",
      "params": {
        "supertrend_periods": [12, 10, 11],
        "supertrend_multipliers": [3.0, 1.0, 2.0],
        "min_trend_agreement": 0.8,
        "timeframe": "15min"
      },
      "trader_params": {
        "stop_loss_pct": 0.02,
        "portfolio_percent_per_trade": 0.5
      }
    },
    {
      "name": "Aggressive_MetaTrend",
      "type": "metatrend",
      "params": {
        "supertrend_periods": [10, 8],
        "supertrend_multipliers": [2.0, 1.0],
        "min_trend_agreement": 0.5,
        "timeframe": "5min"
      },
      "trader_params": {
        "stop_loss_pct": 0.03,
        "portfolio_percent_per_trade": 0.8
      }
    },
    {
      "name": "BBRS_Baseline",
      "type": "bbrs",
      "params": {
        "bb_length": 20,
        "bb_std": 2.0,
        "rsi_length": 14,
        "rsi_overbought": 70,
        "rsi_oversold": 30,
        "timeframe": "15min"
      },
      "trader_params": {
        "stop_loss_pct": 0.025,
        "portfolio_percent_per_trade": 0.6
      }
    }
  ]
}

Command Line Options

• --config: Path to JSON configuration file (required)
• --results-dir: Directory for saving results (default: "results")
• --create-example: Create example config file at specified path
• --verbose: Enable verbose logging for debugging

Error Handling

The runner includes comprehensive error handling:

• Configuration Validation: Validates JSON structure and required fields
• Data File Verification: Checks if data files exist before running
• Strategy Creation: Handles unknown strategy types gracefully
• Backtest Execution: Captures and logs individual strategy failures
• Result Saving: Ensures results are saved even if some strategies fail

Integration

This runner integrates seamlessly with the existing IncrementalTrader framework:

• Uses the same IncBacktester and strategy classes
• Compatible with all existing data formats
• Leverages the same result saving utilities
• Maintains consistency with optimization scripts

Performance

• Sequential Execution: Strategies run one after another for clear logging
• Real-time Results: Individual strategy files saved immediately upon completion
• Efficient Data Loading: Market data loaded once per run for all visualizations
• Progress Tracking: Clear progress indication for long-running backtests
• Detailed Timing: Individual strategy execution times are tracked
• High-Quality Output: Professional 300 DPI plots suitable for presentations

Best Practices

1. Start Small: Test with short date ranges first
2. Validate Data: Ensure data files exist and cover the specified date range
3. Monitor Resources: Watch memory usage for very long backtests
4. Save Configs: Keep configuration files organized for reproducibility
5. Use Descriptive Names: Give strategies clear, descriptive names
6. Test Incrementally: Add strategies one by one when debugging
7. Leverage Visualizations: Use detailed plots to understand market context and strategy behavior
8. Analyze Signals: Review signal CSV files to understand strategy decision patterns
9. Compare Runs: Use organized folder structure to compare different parameter sets
10. Monitor Execution: Watch real-time progress as individual strategies complete