626 lines
18 KiB
Markdown
626 lines
18 KiB
Markdown
# Backtesting Guide
|
||
|
||
This guide explains how to use the IncrementalTrader backtesting framework for comprehensive strategy testing and optimization.
|
||
|
||
## Overview
|
||
|
||
The IncrementalTrader backtesting framework provides:
|
||
- **Single Strategy Testing**: Test individual strategies with detailed metrics
|
||
- **Parameter Optimization**: Systematic parameter sweeps with parallel execution
|
||
- **Performance Analysis**: Comprehensive performance metrics and reporting
|
||
- **Data Management**: Flexible data loading and validation
|
||
- **Result Export**: Multiple output formats for analysis
|
||
|
||
## Quick Start
|
||
|
||
### Basic Backtesting
|
||
|
||
```python
|
||
from IncrementalTrader import IncBacktester, BacktestConfig, MetaTrendStrategy
|
||
|
||
# Configure backtest
|
||
config = BacktestConfig(
|
||
initial_usd=10000,
|
||
stop_loss_pct=0.03,
|
||
take_profit_pct=0.06,
|
||
start_date="2024-01-01",
|
||
end_date="2024-12-31"
|
||
)
|
||
|
||
# Create backtester
|
||
backtester = IncBacktester()
|
||
|
||
# Run single strategy test
|
||
results = backtester.run_single_strategy(
|
||
strategy_class=MetaTrendStrategy,
|
||
strategy_params={"timeframe": "15min"},
|
||
config=config,
|
||
data_file="data/BTCUSDT_1m.csv"
|
||
)
|
||
|
||
# Print results
|
||
print(f"Total Return: {results['performance_metrics']['total_return_pct']:.2f}%")
|
||
print(f"Sharpe Ratio: {results['performance_metrics']['sharpe_ratio']:.2f}")
|
||
print(f"Max Drawdown: {results['performance_metrics']['max_drawdown_pct']:.2f}%")
|
||
```
|
||
|
||
## Configuration
|
||
|
||
### BacktestConfig
|
||
|
||
The main configuration class for backtesting parameters.
|
||
|
||
```python
|
||
from IncrementalTrader import BacktestConfig
|
||
|
||
config = BacktestConfig(
|
||
# Portfolio settings
|
||
initial_usd=10000, # Starting capital
|
||
|
||
# Risk management
|
||
stop_loss_pct=0.03, # 3% stop loss
|
||
take_profit_pct=0.06, # 6% take profit
|
||
|
||
# Time range
|
||
start_date="2024-01-01", # Start date (YYYY-MM-DD)
|
||
end_date="2024-12-31", # End date (YYYY-MM-DD)
|
||
|
||
# Trading settings
|
||
fee_pct=0.001, # 0.1% trading fee
|
||
slippage_pct=0.0005, # 0.05% slippage
|
||
|
||
# Output settings
|
||
output_dir="backtest_results",
|
||
save_trades=True,
|
||
save_portfolio_history=True,
|
||
|
||
# Performance settings
|
||
risk_free_rate=0.02 # 2% annual risk-free rate
|
||
)
|
||
```
|
||
|
||
**Parameters:**
|
||
|
||
| Parameter | Type | Default | Description |
|
||
|-----------|------|---------|-------------|
|
||
| `initial_usd` | float | 10000 | Starting capital in USD |
|
||
| `stop_loss_pct` | float | None | Stop loss percentage (0.03 = 3%) |
|
||
| `take_profit_pct` | float | None | Take profit percentage (0.06 = 6%) |
|
||
| `start_date` | str | None | Start date in YYYY-MM-DD format |
|
||
| `end_date` | str | None | End date in YYYY-MM-DD format |
|
||
| `fee_pct` | float | 0.001 | Trading fee percentage |
|
||
| `slippage_pct` | float | 0.0005 | Slippage percentage |
|
||
| `output_dir` | str | "backtest_results" | Output directory |
|
||
| `save_trades` | bool | True | Save individual trades |
|
||
| `save_portfolio_history` | bool | True | Save portfolio history |
|
||
| `risk_free_rate` | float | 0.02 | Annual risk-free rate for Sharpe ratio |
|
||
|
||
### OptimizationConfig
|
||
|
||
Configuration for parameter optimization.
|
||
|
||
```python
|
||
from IncrementalTrader import OptimizationConfig
|
||
|
||
# Define parameter ranges
|
||
param_ranges = {
|
||
"supertrend_periods": [[10, 20, 30], [15, 25, 35], [20, 30, 40]],
|
||
"supertrend_multipliers": [[2.0, 3.0, 4.0], [1.5, 2.5, 3.5]],
|
||
"min_trend_agreement": [0.5, 0.6, 0.7, 0.8]
|
||
}
|
||
|
||
# Create optimization config
|
||
opt_config = OptimizationConfig(
|
||
base_config=config, # Base BacktestConfig
|
||
param_ranges=param_ranges, # Parameter combinations to test
|
||
max_workers=4, # Number of parallel workers
|
||
optimization_metric="sharpe_ratio", # Metric to optimize
|
||
save_all_results=True # Save all parameter combinations
|
||
)
|
||
```
|
||
|
||
## Single Strategy Testing
|
||
|
||
### Basic Usage
|
||
|
||
```python
|
||
# Test MetaTrend strategy
|
||
results = backtester.run_single_strategy(
|
||
strategy_class=MetaTrendStrategy,
|
||
strategy_params={
|
||
"timeframe": "15min",
|
||
"supertrend_periods": [10, 20, 30],
|
||
"supertrend_multipliers": [2.0, 3.0, 4.0],
|
||
"min_trend_agreement": 0.6
|
||
},
|
||
config=config,
|
||
data_file="data/BTCUSDT_1m.csv"
|
||
)
|
||
```
|
||
|
||
### Results Structure
|
||
|
||
```python
|
||
# Access different result components
|
||
performance = results['performance_metrics']
|
||
trades = results['trades']
|
||
portfolio_history = results['portfolio_history']
|
||
config_used = results['config']
|
||
|
||
# Performance metrics
|
||
print(f"Total Trades: {performance['total_trades']}")
|
||
print(f"Win Rate: {performance['win_rate']:.2f}%")
|
||
print(f"Profit Factor: {performance['profit_factor']:.2f}")
|
||
print(f"Sharpe Ratio: {performance['sharpe_ratio']:.2f}")
|
||
print(f"Sortino Ratio: {performance['sortino_ratio']:.2f}")
|
||
print(f"Max Drawdown: {performance['max_drawdown_pct']:.2f}%")
|
||
print(f"Calmar Ratio: {performance['calmar_ratio']:.2f}")
|
||
|
||
# Trade analysis
|
||
winning_trades = [t for t in trades if t['pnl'] > 0]
|
||
losing_trades = [t for t in trades if t['pnl'] < 0]
|
||
|
||
print(f"Average Win: ${sum(t['pnl'] for t in winning_trades) / len(winning_trades):.2f}")
|
||
print(f"Average Loss: ${sum(t['pnl'] for t in losing_trades) / len(losing_trades):.2f}")
|
||
```
|
||
|
||
### Performance Metrics
|
||
|
||
The backtester calculates comprehensive performance metrics:
|
||
|
||
| Metric | Description | Formula |
|
||
|--------|-------------|---------|
|
||
| Total Return | Overall portfolio return | (Final Value - Initial Value) / Initial Value |
|
||
| Annualized Return | Yearly return rate | (Total Return + 1)^(365/days) - 1 |
|
||
| Volatility | Annualized standard deviation | std(daily_returns) × √365 |
|
||
| Sharpe Ratio | Risk-adjusted return | (Return - Risk Free Rate) / Volatility |
|
||
| Sortino Ratio | Downside risk-adjusted return | (Return - Risk Free Rate) / Downside Deviation |
|
||
| Max Drawdown | Maximum peak-to-trough decline | max((Peak - Trough) / Peak) |
|
||
| Calmar Ratio | Return to max drawdown ratio | Annualized Return / Max Drawdown |
|
||
| Win Rate | Percentage of profitable trades | Winning Trades / Total Trades |
|
||
| Profit Factor | Ratio of gross profit to loss | Gross Profit / Gross Loss |
|
||
|
||
## Parameter Optimization
|
||
|
||
### Basic Optimization
|
||
|
||
```python
|
||
# Define parameter ranges to test
|
||
param_ranges = {
|
||
"timeframe": ["5min", "15min", "30min"],
|
||
"supertrend_periods": [[10, 20, 30], [15, 25, 35]],
|
||
"min_trend_agreement": [0.5, 0.6, 0.7]
|
||
}
|
||
|
||
# Create optimization config
|
||
opt_config = OptimizationConfig(
|
||
base_config=config,
|
||
param_ranges=param_ranges,
|
||
max_workers=4,
|
||
optimization_metric="sharpe_ratio"
|
||
)
|
||
|
||
# Run optimization
|
||
optimization_results = backtester.optimize_strategy(
|
||
strategy_class=MetaTrendStrategy,
|
||
optimization_config=opt_config,
|
||
data_file="data/BTCUSDT_1m.csv"
|
||
)
|
||
|
||
# Get best parameters
|
||
best_params = optimization_results['best_params']
|
||
best_performance = optimization_results['best_performance']
|
||
all_results = optimization_results['all_results']
|
||
|
||
print(f"Best Parameters: {best_params}")
|
||
print(f"Best Sharpe Ratio: {best_performance['sharpe_ratio']:.2f}")
|
||
```
|
||
|
||
### Advanced Optimization
|
||
|
||
```python
|
||
# More complex parameter optimization
|
||
param_ranges = {
|
||
# Strategy parameters
|
||
"timeframe": ["5min", "15min", "30min"],
|
||
"supertrend_periods": [
|
||
[10, 20, 30], [15, 25, 35], [20, 30, 40],
|
||
[10, 15, 20], [25, 35, 45]
|
||
],
|
||
"supertrend_multipliers": [
|
||
[2.0, 3.0, 4.0], [1.5, 2.5, 3.5], [2.5, 3.5, 4.5]
|
||
],
|
||
"min_trend_agreement": [0.4, 0.5, 0.6, 0.7, 0.8],
|
||
|
||
# Risk management (will override config values)
|
||
"stop_loss_pct": [0.02, 0.03, 0.04, 0.05],
|
||
"take_profit_pct": [0.04, 0.06, 0.08, 0.10]
|
||
}
|
||
|
||
# Optimization with custom metric
|
||
def custom_metric(performance):
|
||
"""Custom optimization metric combining return and drawdown."""
|
||
return performance['total_return_pct'] / max(performance['max_drawdown_pct'], 1.0)
|
||
|
||
opt_config = OptimizationConfig(
|
||
base_config=config,
|
||
param_ranges=param_ranges,
|
||
max_workers=8,
|
||
optimization_metric=custom_metric, # Custom function
|
||
save_all_results=True
|
||
)
|
||
|
||
results = backtester.optimize_strategy(
|
||
strategy_class=MetaTrendStrategy,
|
||
optimization_config=opt_config,
|
||
data_file="data/BTCUSDT_1m.csv"
|
||
)
|
||
```
|
||
|
||
### Optimization Metrics
|
||
|
||
You can optimize for different metrics:
|
||
|
||
```python
|
||
# Built-in metrics (string names)
|
||
optimization_metrics = [
|
||
"total_return_pct",
|
||
"sharpe_ratio",
|
||
"sortino_ratio",
|
||
"calmar_ratio",
|
||
"profit_factor",
|
||
"win_rate"
|
||
]
|
||
|
||
# Custom metric function
|
||
def risk_adjusted_return(performance):
|
||
return (performance['total_return_pct'] /
|
||
max(performance['max_drawdown_pct'], 1.0))
|
||
|
||
opt_config = OptimizationConfig(
|
||
base_config=config,
|
||
param_ranges=param_ranges,
|
||
optimization_metric=risk_adjusted_return # Custom function
|
||
)
|
||
```
|
||
|
||
## Data Management
|
||
|
||
### Data Format
|
||
|
||
The backtester expects CSV data with the following columns:
|
||
|
||
```csv
|
||
timestamp,open,high,low,close,volume
|
||
1640995200000,46222.5,46850.0,46150.0,46800.0,1250.5
|
||
1640995260000,46800.0,47000.0,46750.0,46950.0,980.2
|
||
...
|
||
```
|
||
|
||
**Required Columns:**
|
||
- `timestamp`: Unix timestamp in milliseconds
|
||
- `open`: Opening price
|
||
- `high`: Highest price
|
||
- `low`: Lowest price
|
||
- `close`: Closing price
|
||
- `volume`: Trading volume
|
||
|
||
### Data Loading
|
||
|
||
```python
|
||
# The backtester automatically loads and validates data
|
||
results = backtester.run_single_strategy(
|
||
strategy_class=MetaTrendStrategy,
|
||
strategy_params={"timeframe": "15min"},
|
||
config=config,
|
||
data_file="data/BTCUSDT_1m.csv" # Automatically loaded and validated
|
||
)
|
||
|
||
# Data is automatically filtered by start_date and end_date from config
|
||
```
|
||
|
||
### Data Validation
|
||
|
||
The backtester performs automatic data validation:
|
||
|
||
```python
|
||
# Validation checks performed:
|
||
# 1. Required columns present
|
||
# 2. No missing values
|
||
# 3. Timestamps in ascending order
|
||
# 4. Price consistency (high >= low, etc.)
|
||
# 5. Date range filtering
|
||
# 6. Data type validation
|
||
```
|
||
|
||
## Advanced Features
|
||
|
||
### Custom Strategy Testing
|
||
|
||
```python
|
||
# Test your custom strategy
|
||
class MyCustomStrategy(IncStrategyBase):
|
||
def __init__(self, name: str, params: dict = None):
|
||
super().__init__(name, params)
|
||
# Your strategy implementation
|
||
|
||
def _process_aggregated_data(self, timestamp: int, ohlcv: tuple):
|
||
# Your strategy logic
|
||
return IncStrategySignal.HOLD()
|
||
|
||
# Test custom strategy
|
||
results = backtester.run_single_strategy(
|
||
strategy_class=MyCustomStrategy,
|
||
strategy_params={"timeframe": "15min", "custom_param": 42},
|
||
config=config,
|
||
data_file="data/BTCUSDT_1m.csv"
|
||
)
|
||
```
|
||
|
||
### Multiple Strategy Comparison
|
||
|
||
```python
|
||
# Compare different strategies
|
||
strategies_to_test = [
|
||
(MetaTrendStrategy, {"timeframe": "15min"}),
|
||
(BBRSStrategy, {"timeframe": "15min"}),
|
||
(RandomStrategy, {"timeframe": "15min"})
|
||
]
|
||
|
||
comparison_results = {}
|
||
|
||
for strategy_class, params in strategies_to_test:
|
||
results = backtester.run_single_strategy(
|
||
strategy_class=strategy_class,
|
||
strategy_params=params,
|
||
config=config,
|
||
data_file="data/BTCUSDT_1m.csv"
|
||
)
|
||
|
||
strategy_name = strategy_class.__name__
|
||
comparison_results[strategy_name] = results['performance_metrics']
|
||
|
||
# Compare results
|
||
for name, performance in comparison_results.items():
|
||
print(f"{name}:")
|
||
print(f" Return: {performance['total_return_pct']:.2f}%")
|
||
print(f" Sharpe: {performance['sharpe_ratio']:.2f}")
|
||
print(f" Max DD: {performance['max_drawdown_pct']:.2f}%")
|
||
```
|
||
|
||
### Walk-Forward Analysis
|
||
|
||
```python
|
||
# Implement walk-forward analysis
|
||
import pandas as pd
|
||
from datetime import datetime, timedelta
|
||
|
||
def walk_forward_analysis(strategy_class, params, data_file,
|
||
train_months=6, test_months=1):
|
||
"""Perform walk-forward analysis."""
|
||
|
||
# Load full dataset to determine date range
|
||
data = pd.read_csv(data_file)
|
||
data['timestamp'] = pd.to_datetime(data['timestamp'], unit='ms')
|
||
|
||
start_date = data['timestamp'].min()
|
||
end_date = data['timestamp'].max()
|
||
|
||
results = []
|
||
current_date = start_date
|
||
|
||
while current_date + timedelta(days=30*(train_months + test_months)) <= end_date:
|
||
# Define train and test periods
|
||
train_start = current_date
|
||
train_end = current_date + timedelta(days=30*train_months)
|
||
test_start = train_end
|
||
test_end = test_start + timedelta(days=30*test_months)
|
||
|
||
# Optimize on training data
|
||
train_config = BacktestConfig(
|
||
initial_usd=10000,
|
||
start_date=train_start.strftime("%Y-%m-%d"),
|
||
end_date=train_end.strftime("%Y-%m-%d")
|
||
)
|
||
|
||
# Simple parameter optimization (you can expand this)
|
||
best_params = params # In practice, optimize here
|
||
|
||
# Test on out-of-sample data
|
||
test_config = BacktestConfig(
|
||
initial_usd=10000,
|
||
start_date=test_start.strftime("%Y-%m-%d"),
|
||
end_date=test_end.strftime("%Y-%m-%d")
|
||
)
|
||
|
||
test_results = backtester.run_single_strategy(
|
||
strategy_class=strategy_class,
|
||
strategy_params=best_params,
|
||
config=test_config,
|
||
data_file=data_file
|
||
)
|
||
|
||
results.append({
|
||
'test_start': test_start,
|
||
'test_end': test_end,
|
||
'performance': test_results['performance_metrics']
|
||
})
|
||
|
||
# Move to next period
|
||
current_date = test_start
|
||
|
||
return results
|
||
|
||
# Run walk-forward analysis
|
||
wf_results = walk_forward_analysis(
|
||
MetaTrendStrategy,
|
||
{"timeframe": "15min"},
|
||
"data/BTCUSDT_1m.csv"
|
||
)
|
||
|
||
# Analyze walk-forward results
|
||
total_returns = [r['performance']['total_return_pct'] for r in wf_results]
|
||
avg_return = sum(total_returns) / len(total_returns)
|
||
print(f"Average out-of-sample return: {avg_return:.2f}%")
|
||
```
|
||
|
||
## Result Analysis
|
||
|
||
### Detailed Performance Analysis
|
||
|
||
```python
|
||
# Comprehensive result analysis
|
||
def analyze_results(results):
|
||
"""Analyze backtest results in detail."""
|
||
|
||
performance = results['performance_metrics']
|
||
trades = results['trades']
|
||
portfolio_history = results['portfolio_history']
|
||
|
||
print("=== PERFORMANCE SUMMARY ===")
|
||
print(f"Total Return: {performance['total_return_pct']:.2f}%")
|
||
print(f"Annualized Return: {performance['annualized_return_pct']:.2f}%")
|
||
print(f"Volatility: {performance['volatility_pct']:.2f}%")
|
||
print(f"Sharpe Ratio: {performance['sharpe_ratio']:.2f}")
|
||
print(f"Sortino Ratio: {performance['sortino_ratio']:.2f}")
|
||
print(f"Max Drawdown: {performance['max_drawdown_pct']:.2f}%")
|
||
print(f"Calmar Ratio: {performance['calmar_ratio']:.2f}")
|
||
|
||
print("\n=== TRADING STATISTICS ===")
|
||
print(f"Total Trades: {performance['total_trades']}")
|
||
print(f"Win Rate: {performance['win_rate']:.2f}%")
|
||
print(f"Profit Factor: {performance['profit_factor']:.2f}")
|
||
|
||
# Trade analysis
|
||
if trades:
|
||
winning_trades = [t for t in trades if t['pnl'] > 0]
|
||
losing_trades = [t for t in trades if t['pnl'] < 0]
|
||
|
||
if winning_trades:
|
||
avg_win = sum(t['pnl'] for t in winning_trades) / len(winning_trades)
|
||
max_win = max(t['pnl'] for t in winning_trades)
|
||
print(f"Average Win: ${avg_win:.2f}")
|
||
print(f"Largest Win: ${max_win:.2f}")
|
||
|
||
if losing_trades:
|
||
avg_loss = sum(t['pnl'] for t in losing_trades) / len(losing_trades)
|
||
max_loss = min(t['pnl'] for t in losing_trades)
|
||
print(f"Average Loss: ${avg_loss:.2f}")
|
||
print(f"Largest Loss: ${max_loss:.2f}")
|
||
|
||
print("\n=== RISK METRICS ===")
|
||
print(f"Value at Risk (95%): {performance.get('var_95', 'N/A')}")
|
||
print(f"Expected Shortfall (95%): {performance.get('es_95', 'N/A')}")
|
||
|
||
return performance
|
||
|
||
# Analyze results
|
||
performance = analyze_results(results)
|
||
```
|
||
|
||
### Export Results
|
||
|
||
```python
|
||
# Export results to different formats
|
||
def export_results(results, output_dir="backtest_results"):
|
||
"""Export backtest results to files."""
|
||
|
||
import os
|
||
import json
|
||
import pandas as pd
|
||
|
||
os.makedirs(output_dir, exist_ok=True)
|
||
|
||
# Export performance metrics
|
||
with open(f"{output_dir}/performance_metrics.json", 'w') as f:
|
||
json.dump(results['performance_metrics'], f, indent=2)
|
||
|
||
# Export trades
|
||
if results['trades']:
|
||
trades_df = pd.DataFrame(results['trades'])
|
||
trades_df.to_csv(f"{output_dir}/trades.csv", index=False)
|
||
|
||
# Export portfolio history
|
||
if results['portfolio_history']:
|
||
portfolio_df = pd.DataFrame(results['portfolio_history'])
|
||
portfolio_df.to_csv(f"{output_dir}/portfolio_history.csv", index=False)
|
||
|
||
# Export configuration
|
||
config_dict = {
|
||
'initial_usd': results['config'].initial_usd,
|
||
'stop_loss_pct': results['config'].stop_loss_pct,
|
||
'take_profit_pct': results['config'].take_profit_pct,
|
||
'start_date': results['config'].start_date,
|
||
'end_date': results['config'].end_date,
|
||
'fee_pct': results['config'].fee_pct,
|
||
'slippage_pct': results['config'].slippage_pct
|
||
}
|
||
|
||
with open(f"{output_dir}/config.json", 'w') as f:
|
||
json.dump(config_dict, f, indent=2)
|
||
|
||
print(f"Results exported to {output_dir}/")
|
||
|
||
# Export results
|
||
export_results(results)
|
||
```
|
||
|
||
## Best Practices
|
||
|
||
### 1. Data Quality
|
||
|
||
```python
|
||
# Ensure high-quality data
|
||
# - Use clean, validated OHLCV data
|
||
# - Check for gaps and inconsistencies
|
||
# - Use appropriate timeframes for your strategy
|
||
# - Include sufficient history for indicator warmup
|
||
```
|
||
|
||
### 2. Realistic Parameters
|
||
|
||
```python
|
||
# Use realistic trading parameters
|
||
config = BacktestConfig(
|
||
initial_usd=10000,
|
||
fee_pct=0.001, # Realistic trading fees
|
||
slippage_pct=0.0005, # Account for slippage
|
||
stop_loss_pct=0.03, # Reasonable stop loss
|
||
take_profit_pct=0.06 # Reasonable take profit
|
||
)
|
||
```
|
||
|
||
### 3. Overfitting Prevention
|
||
|
||
```python
|
||
# Prevent overfitting
|
||
# - Use out-of-sample testing
|
||
# - Implement walk-forward analysis
|
||
# - Limit parameter optimization ranges
|
||
# - Use cross-validation techniques
|
||
# - Test on multiple time periods and market conditions
|
||
```
|
||
|
||
### 4. Performance Validation
|
||
|
||
```python
|
||
# Validate performance metrics
|
||
# - Check for statistical significance
|
||
# - Analyze trade distribution
|
||
# - Examine drawdown periods
|
||
# - Verify risk-adjusted returns
|
||
# - Compare to benchmarks
|
||
```
|
||
|
||
### 5. Strategy Robustness
|
||
|
||
```python
|
||
# Test strategy robustness
|
||
# - Test on different market conditions
|
||
# - Vary parameter ranges
|
||
# - Check sensitivity to transaction costs
|
||
# - Analyze performance across different timeframes
|
||
# - Test with different data sources
|
||
```
|
||
|
||
This comprehensive backtesting guide provides everything you need to thoroughly test and optimize your trading strategies using the IncrementalTrader framework. Remember that backtesting is just one part of strategy development - always validate results with forward testing before live trading. |