Simon/Cycles
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ok, kind of incremental trading and backtester, but result not alligning

Browse Source
This commit is contained in:
Ajasra
2025-05-27 16:51:43 +08:00
parent ed6d668a8a
commit 8055f46328
29 changed files with 7825 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

460
cycles/IncStrategies/README_BACKTESTER.md Normal file
View File

@@ -0,0 +1,460 @@
# Incremental Backtester
A high-performance backtesting system for incremental trading strategies with multiprocessing support for parameter optimization.
## Overview
The Incremental Backtester provides a complete solution for testing incremental trading strategies:
- **IncTrader**: Manages a single strategy during backtesting
- **IncBacktester**: Orchestrates multiple traders and parameter optimization
- **Multiprocessing Support**: Parallel execution across CPU cores
- **Memory Efficient**: Bounded memory usage regardless of data length
- **Real-time Compatible**: Same interface as live trading systems
## Quick Start
### 1. Basic Single Strategy Backtest
```python
from cycles.IncStrategies import (
IncBacktester, BacktestConfig, IncRandomStrategy
)
# Configure backtest
config = BacktestConfig(
data_file="btc_1min_2023.csv",
start_date="2023-01-01",
end_date="2023-12-31",
initial_usd=10000,
stop_loss_pct=0.02, # 2% stop loss
take_profit_pct=0.05 # 5% take profit
)
# Create strategy
strategy = IncRandomStrategy(params={
"timeframe": "15min",
"entry_probability": 0.1,
"exit_probability": 0.15
})
# Run backtest
backtester = IncBacktester(config)
results = backtester.run_single_strategy(strategy)
print(f"Profit: {results['profit_ratio']*100:.2f}%")
print(f"Trades: {results['n_trades']}")
print(f"Win Rate: {results['win_rate']*100:.1f}%")
```
### 2. Multiple Strategies
```python
strategies = [
IncRandomStrategy(params={"timeframe": "15min"}),
IncRandomStrategy(params={"timeframe": "30min"}),
IncMetaTrendStrategy(params={"timeframe": "15min"})
]
results = backtester.run_multiple_strategies(strategies)
for result in results:
print(f"{result['strategy_name']}: {result['profit_ratio']*100:.2f}%")
```
### 3. Parameter Optimization
```python
# Define parameter grids
strategy_param_grid = {
"timeframe": ["15min", "30min", "1h"],
"entry_probability": [0.05, 0.1, 0.15],
"exit_probability": [0.1, 0.15, 0.2]
}
trader_param_grid = {
"stop_loss_pct": [0.01, 0.02, 0.03],
"take_profit_pct": [0.03, 0.05, 0.07]
}
# Run optimization (uses all CPU cores)
results = backtester.optimize_parameters(
strategy_class=IncRandomStrategy,
param_grid=strategy_param_grid,
trader_param_grid=trader_param_grid,
max_workers=8 # Use 8 CPU cores
)
# Get summary statistics
summary = backtester.get_summary_statistics(results)
print(f"Best profit: {summary['profit_ratio']['max']*100:.2f}%")
# Save results
backtester.save_results(results, "optimization_results.csv")
```
## Architecture
### IncTrader Class
Manages a single strategy during backtesting:
```python
trader = IncTrader(
strategy=strategy,
initial_usd=10000,
params={
"stop_loss_pct": 0.02,
"take_profit_pct": 0.05
}
)
# Process data sequentially
for timestamp, ohlcv_data in data_stream:
trader.process_data_point(timestamp, ohlcv_data)
# Get results
results = trader.get_results()
```
**Key Features:**
- Position management (USD/coin balance)
- Trade execution based on strategy signals
- Stop loss and take profit handling
- Performance tracking and metrics
- Fee calculation using existing MarketFees
### IncBacktester Class
Orchestrates multiple traders and handles data loading:
```python
backtester = IncBacktester(config, storage)
# Single strategy
results = backtester.run_single_strategy(strategy)
# Multiple strategies
results = backtester.run_multiple_strategies(strategies)
# Parameter optimization
results = backtester.optimize_parameters(strategy_class, param_grid)
```
**Key Features:**
- Data loading using existing Storage class
- Multiprocessing for parameter optimization
- Result aggregation and analysis
- Summary statistics calculation
- CSV export functionality
### BacktestConfig Class
Configuration for backtesting runs:
```python
config = BacktestConfig(
data_file="btc_1min_2023.csv",
start_date="2023-01-01",
end_date="2023-12-31",
initial_usd=10000,
timeframe="1min",
# Trader parameters
stop_loss_pct=0.02,
take_profit_pct=0.05,
# Performance settings
max_workers=None, # Auto-detect CPU cores
chunk_size=1000
)
```
## Data Requirements
### Input Data Format
The backtester expects minute-level OHLCV data in CSV format:
```csv
timestamp,open,high,low,close,volume
1672531200,16625.1,16634.5,16620.0,16628.3,125.45
1672531260,16628.3,16635.2,16625.8,16631.7,98.32
...
```
**Requirements:**
- Timestamp column (Unix timestamp or datetime)
- OHLCV columns: open, high, low, close, volume
- Minute-level frequency (strategies handle timeframe aggregation)
- Sorted by timestamp (ascending)
### Data Loading
Uses the existing Storage class for data loading:
```python
from cycles.utils.storage import Storage
storage = Storage()
data = storage.load_data(
"btc_1min_2023.csv",
"2023-01-01",
"2023-12-31"
)
```
## Performance Features
### Multiprocessing Support
Parameter optimization automatically distributes work across CPU cores:
```python
# Automatic CPU detection
results = backtester.optimize_parameters(strategy_class, param_grid)
# Manual worker count
results = backtester.optimize_parameters(
strategy_class, param_grid, max_workers=4
)
# Single-threaded (for debugging)
results = backtester.optimize_parameters(
strategy_class, param_grid, max_workers=1
)
```
### Memory Efficiency
- **Bounded Memory**: Strategy buffers have fixed size limits
- **Incremental Processing**: No need to load entire datasets into memory
- **Efficient Data Structures**: Optimized for sequential processing
### Performance Monitoring
Built-in performance tracking:
```python
results = backtester.run_single_strategy(strategy)
print(f"Backtest duration: {results['backtest_duration_seconds']:.2f}s")
print(f"Data points processed: {results['data_points_processed']}")
print(f"Processing rate: {results['data_points']/results['backtest_duration_seconds']:.0f} points/sec")
```
## Result Analysis
### Individual Results
Each backtest returns comprehensive metrics:
```python
{
"strategy_name": "IncRandomStrategy",
"strategy_params": {"timeframe": "15min", ...},
"trader_params": {"stop_loss_pct": 0.02, ...},
"initial_usd": 10000.0,
"final_usd": 10250.0,
"profit_ratio": 0.025,
"n_trades": 15,
"win_rate": 0.6,
"max_drawdown": 0.08,
"avg_trade": 0.0167,
"total_fees_usd": 45.32,
"trades": [...], # Individual trade records
"backtest_duration_seconds": 2.45
}
```
### Summary Statistics
For parameter optimization runs:
```python
summary = backtester.get_summary_statistics(results)
{
"total_runs": 108,
"successful_runs": 105,
"failed_runs": 3,
"profit_ratio": {
"mean": 0.023,
"std": 0.045,
"min": -0.12,
"max": 0.18,
"median": 0.019
},
"best_run": {...},
"worst_run": {...}
}
```
### Export Results
Save results to CSV for further analysis:
```python
backtester.save_results(results, "backtest_results.csv")
```
Output includes:
- Strategy and trader parameters
- Performance metrics
- Trade statistics
- Execution timing
## Integration with Existing System
### Compatibility
The incremental backtester integrates seamlessly with existing components:
- **Storage Class**: Uses existing data loading infrastructure
- **MarketFees**: Uses existing fee calculation
- **Strategy Interface**: Compatible with incremental strategies
- **Result Format**: Similar to existing Backtest class
### Migration from Original Backtester
```python
# Original backtester
from cycles.backtest import Backtest
# Incremental backtester
from cycles.IncStrategies import IncBacktester, BacktestConfig
# Similar interface, enhanced capabilities
config = BacktestConfig(...)
backtester = IncBacktester(config)
results = backtester.run_single_strategy(strategy)
```
## Testing
### Synthetic Data Testing
Test with synthetic data before using real market data:
```python
from cycles.IncStrategies.test_inc_backtester import main
# Run all tests
main()
```
### Unit Tests
Individual component testing:
```python
# Test IncTrader
from cycles.IncStrategies.test_inc_backtester import test_inc_trader
test_inc_trader()
# Test IncBacktester
from cycles.IncStrategies.test_inc_backtester import test_inc_backtester
test_inc_backtester()
```
## Examples
See `example_backtest.py` for comprehensive usage examples:
```python
from cycles.IncStrategies.example_backtest import (
example_single_strategy_backtest,
example_parameter_optimization,
example_custom_analysis
)
# Run examples
example_single_strategy_backtest()
example_parameter_optimization()
```
## Best Practices
### 1. Data Preparation
- Ensure data quality (no gaps, correct format)
- Use appropriate date ranges for testing
- Consider market conditions in test periods
### 2. Parameter Optimization
- Start with small parameter grids for testing
- Use representative time periods
- Consider overfitting risks
- Validate results on out-of-sample data
### 3. Performance Optimization
- Use multiprocessing for large parameter grids
- Monitor memory usage for long backtests
- Profile bottlenecks for optimization
### 4. Result Validation
- Compare with original backtester for validation
- Check trade logic manually for small samples
- Verify fee calculations and position management
## Troubleshooting
### Common Issues
1. **Data Loading Errors**
- Check file path and format
- Verify date range availability
- Ensure required columns exist
2. **Strategy Errors**
- Check strategy initialization
- Verify parameter validity
- Monitor warmup period completion
3. **Performance Issues**
- Reduce parameter grid size
- Limit worker count for memory constraints
- Use shorter time periods for testing
### Debug Mode
Enable detailed logging:
```python
import logging
logging.basicConfig(level=logging.DEBUG)
# Run with detailed output
results = backtester.run_single_strategy(strategy)
```
### Memory Monitoring
Monitor memory usage during optimization:
```python
import psutil
import os
process = psutil.Process(os.getpid())
print(f"Memory usage: {process.memory_info().rss / 1024 / 1024:.1f} MB")
```
## Future Enhancements
- **Live Trading Integration**: Direct connection to trading systems
- **Advanced Analytics**: Risk metrics, Sharpe ratio, etc.
- **Visualization**: Built-in plotting and analysis tools
- **Database Support**: Direct database connectivity
- **Strategy Combinations**: Multi-strategy portfolio testing
## Support
For issues and questions:
1. Check the test scripts for working examples
2. Review the TODO.md for known limitations
3. Examine the base strategy implementations
4. Use debug logging for detailed troubleshooting

19
cycles/IncStrategies/__init__.py
View File

@@ -17,11 +17,17 @@ Classes:
IncDefaultStrategy: Incremental implementation of the default Supertrend strategy
IncBBRSStrategy: Incremental implementation of Bollinger Bands + RSI strategy
IncStrategyManager: Manager for coordinating multiple incremental strategies
IncTrader: Trader that manages a single strategy during backtesting
IncBacktester: Backtester for testing incremental strategies with multiprocessing
BacktestConfig: Configuration class for backtesting runs
"""
from .base import IncStrategyBase, IncStrategySignal
from .random_strategy import IncRandomStrategy
from .metatrend_strategy import IncMetaTrendStrategy, MetaTrendStrategy
from .inc_trader import IncTrader, TradeRecord
from .inc_backtester import IncBacktester, BacktestConfig
# Note: These will be implemented in subsequent phases
# from .default_strategy import IncDefaultStrategy
@@ -38,12 +44,25 @@ AVAILABLE_STRATEGIES = {
}
__all__ = [
# Base classes
'IncStrategyBase',
'IncStrategySignal',
# Strategies
'IncRandomStrategy',
'IncMetaTrendStrategy',
'MetaTrendStrategy',
# Backtesting components
'IncTrader',
'IncBacktester',
'BacktestConfig',
'TradeRecord',
# Registry
'AVAILABLE_STRATEGIES'
# Future implementations
# 'IncDefaultStrategy',
# 'IncBBRSStrategy',
# 'IncStrategyManager'

20
cycles/IncStrategies/base.py
View File

@@ -86,8 +86,24 @@ class TimeframeAggregator:
return None # No completed bar yet
def _get_bar_start_time(self, timestamp: pd.Timestamp) -> pd.Timestamp:
"""Calculate the start time of the timeframe bar for given timestamp."""
# Round down to the nearest timeframe boundary
"""Calculate the start time of the timeframe bar for given timestamp.
This method now aligns with pandas resampling to ensure consistency
with the original strategy's bar boundaries.
"""
# Use pandas-style resampling alignment
# This ensures bars align to standard boundaries (e.g., 00:00, 00:15, 00:30, 00:45)
freq_str = f'{self.timeframe_minutes}min'
# Create a temporary series with the timestamp and resample to get the bar start
temp_series = pd.Series([1], index=[timestamp])
resampled = temp_series.resample(freq_str)
# Get the first group's name (which is the bar start time)
for bar_start, _ in resampled:
return bar_start
# Fallback to original method if resampling fails
minutes_since_midnight = timestamp.hour * 60 + timestamp.minute
bar_minutes = (minutes_since_midnight // self.timeframe_minutes) * self.timeframe_minutes

447
cycles/IncStrategies/example_backtest.py Normal file
View File

@@ -0,0 +1,447 @@
"""
Example usage of the Incremental Backtester.
This script demonstrates how to use the IncBacktester for various scenarios:
1. Single strategy backtesting
2. Multiple strategy comparison
3. Parameter optimization with multiprocessing
4. Custom analysis and result saving
5. Comprehensive result logging and action tracking
Run this script to see the backtester in action with real or synthetic data.
"""
import pandas as pd
import numpy as np
import logging
from datetime import datetime, timedelta
import os
from cycles.IncStrategies import (
IncBacktester, BacktestConfig, IncRandomStrategy
)
from cycles.utils.storage import Storage
# Configure logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
def ensure_results_directory():
"""Ensure the results directory exists."""
results_dir = "results"
if not os.path.exists(results_dir):
os.makedirs(results_dir)
logger.info(f"Created results directory: {results_dir}")
return results_dir
def create_sample_data(days: int = 30) -> pd.DataFrame:
"""
Create sample OHLCV data for demonstration.
Args:
days: Number of days of data to generate
Returns:
pd.DataFrame: Sample OHLCV data
"""
# Create date range
end_date = datetime.now()
start_date = end_date - timedelta(days=days)
timestamps = pd.date_range(start=start_date, end=end_date, freq='1min')
# Generate realistic price data
np.random.seed(42)
n_points = len(timestamps)
# Start with a base price
base_price = 45000
# Generate price movements with trend and volatility
trend = np.linspace(0, 0.1, n_points) # Slight upward trend
volatility = np.random.normal(0, 0.002, n_points) # 0.2% volatility
# Calculate prices
log_returns = trend + volatility
prices = base_price * np.exp(np.cumsum(log_returns))
# Generate OHLCV data
data = []
for i, (timestamp, close_price) in enumerate(zip(timestamps, prices)):
# Generate realistic OHLC
intrabar_vol = close_price * 0.001
open_price = close_price + np.random.normal(0, intrabar_vol)
high_price = max(open_price, close_price) + abs(np.random.normal(0, intrabar_vol))
low_price = min(open_price, close_price) - abs(np.random.normal(0, intrabar_vol))
volume = np.random.uniform(50, 500)
data.append({
'open': open_price,
'high': high_price,
'low': low_price,
'close': close_price,
'volume': volume
})
df = pd.DataFrame(data, index=timestamps)
return df
def example_single_strategy():
"""Example 1: Single strategy backtesting with comprehensive results."""
print("\n" + "="*60)
print("EXAMPLE 1: Single Strategy Backtesting")
print("="*60)
# Create sample data
data = create_sample_data(days=7) # 1 week of data
# Save data
storage = Storage()
data_file = "sample_data_single.csv"
storage.save_data(data, data_file)
# Configure backtest
config = BacktestConfig(
data_file=data_file,
start_date=data.index[0].strftime("%Y-%m-%d"),
end_date=data.index[-1].strftime("%Y-%m-%d"),
initial_usd=10000,
stop_loss_pct=0.02,
take_profit_pct=0.05
)
# Create strategy
strategy = IncRandomStrategy(params={
"timeframe": "15min",
"entry_probability": 0.15,
"exit_probability": 0.2,
"random_seed": 42
})
# Run backtest
backtester = IncBacktester(config, storage)
results = backtester.run_single_strategy(strategy)
# Print results
print(f"\nResults:")
print(f" Strategy: {results['strategy_name']}")
print(f" Profit: {results['profit_ratio']*100:.2f}%")
print(f" Final Balance: ${results['final_usd']:,.2f}")
print(f" Trades: {results['n_trades']}")
print(f" Win Rate: {results['win_rate']*100:.1f}%")
print(f" Max Drawdown: {results['max_drawdown']*100:.2f}%")
# Save comprehensive results
backtester.save_comprehensive_results([results], "example_single_strategy")
# Cleanup
if os.path.exists(f"data/{data_file}"):
os.remove(f"data/{data_file}")
return results
def example_multiple_strategies():
"""Example 2: Multiple strategy comparison with comprehensive results."""
print("\n" + "="*60)
print("EXAMPLE 2: Multiple Strategy Comparison")
print("="*60)
# Create sample data
data = create_sample_data(days=10) # 10 days of data
# Save data
storage = Storage()
data_file = "sample_data_multiple.csv"
storage.save_data(data, data_file)
# Configure backtest
config = BacktestConfig(
data_file=data_file,
start_date=data.index[0].strftime("%Y-%m-%d"),
end_date=data.index[-1].strftime("%Y-%m-%d"),
initial_usd=10000,
stop_loss_pct=0.015
)
# Create multiple strategies with different parameters
strategies = [
IncRandomStrategy(params={
"timeframe": "5min",
"entry_probability": 0.1,
"exit_probability": 0.15,
"random_seed": 42
}),
IncRandomStrategy(params={
"timeframe": "15min",
"entry_probability": 0.12,
"exit_probability": 0.18,
"random_seed": 123
}),
IncRandomStrategy(params={
"timeframe": "30min",
"entry_probability": 0.08,
"exit_probability": 0.12,
"random_seed": 456
}),
IncRandomStrategy(params={
"timeframe": "1h",
"entry_probability": 0.06,
"exit_probability": 0.1,
"random_seed": 789
})
]
# Run backtest
backtester = IncBacktester(config, storage)
results = backtester.run_multiple_strategies(strategies)
# Print comparison
print(f"\nStrategy Comparison:")
print(f"{'Strategy':<20} {'Timeframe':<10} {'Profit %':<10} {'Trades':<8} {'Win Rate %':<12}")
print("-" * 70)
for i, result in enumerate(results):
if result.get("success", True):
timeframe = result['strategy_params']['timeframe']
profit = result['profit_ratio'] * 100
trades = result['n_trades']
win_rate = result['win_rate'] * 100
print(f"Strategy {i+1:<13} {timeframe:<10} {profit:<10.2f} {trades:<8} {win_rate:<12.1f}")
# Get summary statistics
summary = backtester.get_summary_statistics(results)
print(f"\nSummary Statistics:")
print(f" Best Profit: {summary['profit_ratio']['max']*100:.2f}%")
print(f" Worst Profit: {summary['profit_ratio']['min']*100:.2f}%")
print(f" Average Profit: {summary['profit_ratio']['mean']*100:.2f}%")
print(f" Profit Std Dev: {summary['profit_ratio']['std']*100:.2f}%")
# Save comprehensive results
backtester.save_comprehensive_results(results, "example_multiple_strategies", summary)
# Cleanup
if os.path.exists(f"data/{data_file}"):
os.remove(f"data/{data_file}")
return results, summary
def example_parameter_optimization():
"""Example 3: Parameter optimization with multiprocessing and comprehensive results."""
print("\n" + "="*60)
print("EXAMPLE 3: Parameter Optimization")
print("="*60)
# Create sample data
data = create_sample_data(days=5) # 5 days for faster optimization
# Save data
storage = Storage()
data_file = "sample_data_optimization.csv"
storage.save_data(data, data_file)
# Configure backtest
config = BacktestConfig(
data_file=data_file,
start_date=data.index[0].strftime("%Y-%m-%d"),
end_date=data.index[-1].strftime("%Y-%m-%d"),
initial_usd=10000
)
# Define parameter grids
strategy_param_grid = {
"timeframe": ["5min", "15min", "30min"],
"entry_probability": [0.08, 0.12, 0.16],
"exit_probability": [0.1, 0.15, 0.2],
"random_seed": [42] # Keep seed constant for fair comparison
}
trader_param_grid = {
"stop_loss_pct": [0.01, 0.015, 0.02],
"take_profit_pct": [0.0, 0.03, 0.05]
}
# Run optimization (will use SystemUtils to determine optimal workers)
backtester = IncBacktester(config, storage)
print(f"Starting optimization with {len(strategy_param_grid['timeframe']) * len(strategy_param_grid['entry_probability']) * len(strategy_param_grid['exit_probability']) * len(trader_param_grid['stop_loss_pct']) * len(trader_param_grid['take_profit_pct'])} combinations...")
results = backtester.optimize_parameters(
strategy_class=IncRandomStrategy,
param_grid=strategy_param_grid,
trader_param_grid=trader_param_grid,
max_workers=None # Use SystemUtils for optimal worker count
)
# Get summary
summary = backtester.get_summary_statistics(results)
# Print optimization results
print(f"\nOptimization Results:")
print(f" Total Combinations: {summary['total_runs']}")
print(f" Successful Runs: {summary['successful_runs']}")
print(f" Failed Runs: {summary['failed_runs']}")
if summary['successful_runs'] > 0:
print(f" Best Profit: {summary['profit_ratio']['max']*100:.2f}%")
print(f" Worst Profit: {summary['profit_ratio']['min']*100:.2f}%")
print(f" Average Profit: {summary['profit_ratio']['mean']*100:.2f}%")
# Show top 3 configurations
valid_results = [r for r in results if r.get("success", True)]
valid_results.sort(key=lambda x: x["profit_ratio"], reverse=True)
print(f"\nTop 3 Configurations:")
for i, result in enumerate(valid_results[:3]):
print(f" {i+1}. Profit: {result['profit_ratio']*100:.2f}% | "
f"Timeframe: {result['strategy_params']['timeframe']} | "
f"Entry Prob: {result['strategy_params']['entry_probability']} | "
f"Stop Loss: {result['trader_params']['stop_loss_pct']*100:.1f}%")
# Save comprehensive results
backtester.save_comprehensive_results(results, "example_parameter_optimization", summary)
# Cleanup
if os.path.exists(f"data/{data_file}"):
os.remove(f"data/{data_file}")
return results, summary
def example_custom_analysis():
"""Example 4: Custom analysis with detailed result examination."""
print("\n" + "="*60)
print("EXAMPLE 4: Custom Analysis")
print("="*60)
# Create sample data with more volatility for interesting results
data = create_sample_data(days=14) # 2 weeks
# Save data
storage = Storage()
data_file = "sample_data_analysis.csv"
storage.save_data(data, data_file)
# Configure backtest
config = BacktestConfig(
data_file=data_file,
start_date=data.index[0].strftime("%Y-%m-%d"),
end_date=data.index[-1].strftime("%Y-%m-%d"),
initial_usd=25000, # Larger starting capital
stop_loss_pct=0.025,
take_profit_pct=0.04
)
# Create strategy with specific parameters for analysis
strategy = IncRandomStrategy(params={
"timeframe": "30min",
"entry_probability": 0.1,
"exit_probability": 0.15,
"random_seed": 42
})
# Run backtest
backtester = IncBacktester(config, storage)
results = backtester.run_single_strategy(strategy)
# Detailed analysis
print(f"\nDetailed Analysis:")
print(f" Strategy: {results['strategy_name']}")
print(f" Timeframe: {results['strategy_params']['timeframe']}")
print(f" Data Period: {config.start_date} to {config.end_date}")
print(f" Data Points: {results['data_points']:,}")
print(f" Processing Time: {results['backtest_duration_seconds']:.2f}s")
print(f"\nPerformance Metrics:")
print(f" Initial Capital: ${results['initial_usd']:,.2f}")
print(f" Final Balance: ${results['final_usd']:,.2f}")
print(f" Total Return: {results['profit_ratio']*100:.2f}%")
print(f" Total Trades: {results['n_trades']}")
if results['n_trades'] > 0:
print(f" Win Rate: {results['win_rate']*100:.1f}%")
print(f" Average Trade: ${results['avg_trade']:.2f}")
print(f" Max Drawdown: {results['max_drawdown']*100:.2f}%")
print(f" Total Fees: ${results['total_fees_usd']:.2f}")
# Calculate additional metrics
days_traded = (pd.to_datetime(config.end_date) - pd.to_datetime(config.start_date)).days
annualized_return = (1 + results['profit_ratio']) ** (365 / days_traded) - 1
print(f" Annualized Return: {annualized_return*100:.2f}%")
# Risk metrics
if results['max_drawdown'] > 0:
calmar_ratio = annualized_return / results['max_drawdown']
print(f" Calmar Ratio: {calmar_ratio:.2f}")
# Save comprehensive results with custom analysis
backtester.save_comprehensive_results([results], "example_custom_analysis")
# Cleanup
if os.path.exists(f"data/{data_file}"):
os.remove(f"data/{data_file}")
return results
def main():
"""Run all examples."""
print("Incremental Backtester Examples")
print("="*60)
print("This script demonstrates various features of the IncBacktester:")
print("1. Single strategy backtesting")
print("2. Multiple strategy comparison")
print("3. Parameter optimization with multiprocessing")
print("4. Custom analysis and metrics")
print("5. Comprehensive result saving and action logging")
# Ensure results directory exists
ensure_results_directory()
try:
# Run all examples
single_results = example_single_strategy()
multiple_results, multiple_summary = example_multiple_strategies()
optimization_results, optimization_summary = example_parameter_optimization()
analysis_results = example_custom_analysis()
print("\n" + "="*60)
print("ALL EXAMPLES COMPLETED SUCCESSFULLY!")
print("="*60)
print("\n📊 Comprehensive results have been saved to the 'results' directory.")
print("Each example generated multiple files:")
print(" 📋 Summary JSON with session info and statistics")
print(" 📈 Detailed CSV with all backtest results")
print(" 📝 Action log JSON with all operations performed")
print(" 📁 Individual strategy JSON files with trades and details")
print(" 🗂️ Master index JSON for easy navigation")
print(f"\n🎯 Key Insights:")
print(f" • Single strategy achieved {single_results['profit_ratio']*100:.2f}% return")
print(f" • Multiple strategies: best {multiple_summary['profit_ratio']['max']*100:.2f}%, worst {multiple_summary['profit_ratio']['min']*100:.2f}%")
print(f" • Optimization tested {optimization_summary['total_runs']} combinations")
print(f" • Custom analysis provided detailed risk metrics")
print(f"\n🔧 System Performance:")
print(f" • Used SystemUtils for optimal CPU core utilization")
print(f" • All actions logged for reproducibility")
print(f" • Results saved in multiple formats for analysis")
print(f"\n✅ The incremental backtester is ready for production use!")
except Exception as e:
logger.error(f"Example failed: {e}")
print(f"\nError: {e}")
import traceback
traceback.print_exc()
if __name__ == "__main__":
main()

736
cycles/IncStrategies/inc_backtester.py Normal file
View File

@@ -0,0 +1,736 @@
"""
Incremental Backtester for testing incremental strategies.
This module provides the IncBacktester class that orchestrates multiple IncTraders
for parallel testing, handles data loading and feeding, and supports multiprocessing
for parameter optimization.
"""
import pandas as pd
import numpy as np
from typing import Dict, List, Optional, Any, Callable, Union, Tuple
import logging
import time
from concurrent.futures import ProcessPoolExecutor, as_completed
from itertools import product
import multiprocessing as mp
from dataclasses import dataclass
import json
from datetime import datetime
from .inc_trader import IncTrader
from .base import IncStrategyBase
from ..utils.storage import Storage
from ..utils.system import SystemUtils
logger = logging.getLogger(__name__)
def _worker_function(args: Tuple[type, Dict, Dict, 'BacktestConfig', str]) -> Dict[str, Any]:
"""
Worker function for multiprocessing parameter optimization.
This function must be at module level to be picklable for multiprocessing.
Args:
args: Tuple containing (strategy_class, strategy_params, trader_params, config, data_file)
Returns:
Dict containing backtest results
"""
try:
strategy_class, strategy_params, trader_params, config, data_file = args
# Create new storage and backtester instance for this worker
storage = Storage()
worker_backtester = IncBacktester(config, storage)
# Create strategy instance
strategy = strategy_class(params=strategy_params)
# Run backtest
result = worker_backtester.run_single_strategy(strategy, trader_params)
result["success"] = True
return result
except Exception as e:
logger.error(f"Worker error for {strategy_params}, {trader_params}: {e}")
return {
"strategy_params": strategy_params,
"trader_params": trader_params,
"error": str(e),
"success": False
}
@dataclass
class BacktestConfig:
"""Configuration for backtesting runs."""
data_file: str
start_date: str
end_date: str
initial_usd: float = 10000
timeframe: str = "1min"
# Trader parameters
stop_loss_pct: float = 0.0
take_profit_pct: float = 0.0
# Performance settings
max_workers: Optional[int] = None
chunk_size: int = 1000
class IncBacktester:
"""
Incremental backtester for testing incremental strategies.
This class orchestrates multiple IncTraders for parallel testing:
- Loads data using the existing Storage class
- Creates multiple IncTrader instances with different parameters
- Feeds data sequentially to all traders
- Collects and aggregates results
- Supports multiprocessing for parallel execution
- Uses SystemUtils for optimal worker count determination
The backtester can run multiple strategies simultaneously or test
parameter combinations across multiple CPU cores.
Example:
# Single strategy backtest
config = BacktestConfig(
data_file="btc_1min_2023.csv",
start_date="2023-01-01",
end_date="2023-12-31",
initial_usd=10000
)
strategy = IncRandomStrategy(params={"timeframe": "15min"})
backtester = IncBacktester(config)
results = backtester.run_single_strategy(strategy)
# Multiple strategies
strategies = [strategy1, strategy2, strategy3]
results = backtester.run_multiple_strategies(strategies)
# Parameter optimization
param_grid = {
"timeframe": ["5min", "15min", "30min"],
"stop_loss_pct": [0.01, 0.02, 0.03]
}
results = backtester.optimize_parameters(strategy_class, param_grid)
"""
def __init__(self, config: BacktestConfig, storage: Optional[Storage] = None):
"""
Initialize the incremental backtester.
Args:
config: Backtesting configuration
storage: Storage instance for data loading (creates new if None)
"""
self.config = config
self.storage = storage or Storage()
self.system_utils = SystemUtils(logging=logger)
self.data = None
self.results_cache = {}
# Track all actions performed during backtesting
self.action_log = []
self.session_start_time = datetime.now()
logger.info(f"IncBacktester initialized: {config.data_file}, "
f"{config.start_date} to {config.end_date}")
self._log_action("backtester_initialized", {
"config": config.__dict__,
"session_start": self.session_start_time.isoformat()
})
def _log_action(self, action_type: str, details: Dict[str, Any]) -> None:
"""Log an action performed during backtesting."""
self.action_log.append({
"timestamp": datetime.now().isoformat(),
"action_type": action_type,
"details": details
})
def load_data(self) -> pd.DataFrame:
"""
Load and prepare data for backtesting.
Returns:
pd.DataFrame: Loaded OHLCV data with DatetimeIndex
"""
if self.data is None:
logger.info(f"Loading data from {self.config.data_file}...")
start_time = time.time()
self.data = self.storage.load_data(
self.config.data_file,
self.config.start_date,
self.config.end_date
)
load_time = time.time() - start_time
logger.info(f"Data loaded: {len(self.data)} rows in {load_time:.2f}s")
# Validate data
if self.data.empty:
raise ValueError(f"No data loaded for the specified date range")
required_columns = ['open', 'high', 'low', 'close', 'volume']
missing_columns = [col for col in required_columns if col not in self.data.columns]
if missing_columns:
raise ValueError(f"Missing required columns: {missing_columns}")
self._log_action("data_loaded", {
"file": self.config.data_file,
"rows": len(self.data),
"load_time_seconds": load_time,
"date_range": f"{self.config.start_date} to {self.config.end_date}",
"columns": list(self.data.columns)
})
return self.data
def run_single_strategy(self, strategy: IncStrategyBase,
trader_params: Optional[Dict] = None) -> Dict[str, Any]:
"""
Run backtest for a single strategy.
Args:
strategy: Incremental strategy instance
trader_params: Additional trader parameters
Returns:
Dict containing backtest results
"""
data = self.load_data()
# Merge trader parameters
final_trader_params = {
"stop_loss_pct": self.config.stop_loss_pct,
"take_profit_pct": self.config.take_profit_pct
}
if trader_params:
final_trader_params.update(trader_params)
# Create trader
trader = IncTrader(
strategy=strategy,
initial_usd=self.config.initial_usd,
params=final_trader_params
)
# Run backtest
logger.info(f"Starting backtest for {strategy.name}...")
start_time = time.time()
self._log_action("single_strategy_backtest_started", {
"strategy_name": strategy.name,
"strategy_params": strategy.params,
"trader_params": final_trader_params,
"data_points": len(data)
})
for timestamp, row in data.iterrows():
ohlcv_data = {
'open': row['open'],
'high': row['high'],
'low': row['low'],
'close': row['close'],
'volume': row['volume']
}
trader.process_data_point(timestamp, ohlcv_data)
# Finalize and get results
trader.finalize()
results = trader.get_results()
backtest_time = time.time() - start_time
results["backtest_duration_seconds"] = backtest_time
results["data_points"] = len(data)
results["config"] = self.config.__dict__
logger.info(f"Backtest completed for {strategy.name} in {backtest_time:.2f}s: "
f"${results['final_usd']:.2f} ({results['profit_ratio']*100:.2f}%), "
f"{results['n_trades']} trades")
self._log_action("single_strategy_backtest_completed", {
"strategy_name": strategy.name,
"backtest_duration_seconds": backtest_time,
"final_usd": results['final_usd'],
"profit_ratio": results['profit_ratio'],
"n_trades": results['n_trades'],
"win_rate": results['win_rate']
})
return results
def run_multiple_strategies(self, strategies: List[IncStrategyBase],
trader_params: Optional[Dict] = None) -> List[Dict[str, Any]]:
"""
Run backtest for multiple strategies simultaneously.
Args:
strategies: List of incremental strategy instances
trader_params: Additional trader parameters
Returns:
List of backtest results for each strategy
"""
self._log_action("multiple_strategies_backtest_started", {
"strategy_count": len(strategies),
"strategy_names": [s.name for s in strategies]
})
results = []
for strategy in strategies:
try:
result = self.run_single_strategy(strategy, trader_params)
results.append(result)
except Exception as e:
logger.error(f"Error running strategy {strategy.name}: {e}")
# Add error result
error_result = {
"strategy_name": strategy.name,
"error": str(e),
"success": False
}
results.append(error_result)
self._log_action("strategy_error", {
"strategy_name": strategy.name,
"error": str(e)
})
self._log_action("multiple_strategies_backtest_completed", {
"total_strategies": len(strategies),
"successful_strategies": len([r for r in results if r.get("success", True)]),
"failed_strategies": len([r for r in results if not r.get("success", True)])
})
return results
def optimize_parameters(self, strategy_class: type, param_grid: Dict[str, List],
trader_param_grid: Optional[Dict[str, List]] = None,
max_workers: Optional[int] = None) -> List[Dict[str, Any]]:
"""
Optimize strategy parameters using grid search with multiprocessing.
Args:
strategy_class: Strategy class to instantiate
param_grid: Grid of strategy parameters to test
trader_param_grid: Grid of trader parameters to test
max_workers: Maximum number of worker processes (uses SystemUtils if None)
Returns:
List of results for each parameter combination
"""
# Generate parameter combinations
strategy_combinations = list(self._generate_param_combinations(param_grid))
trader_combinations = list(self._generate_param_combinations(trader_param_grid or {}))
# If no trader param grid, use default
if not trader_combinations:
trader_combinations = [{}]
# Create all combinations
all_combinations = []
for strategy_params in strategy_combinations:
for trader_params in trader_combinations:
all_combinations.append((strategy_params, trader_params))
logger.info(f"Starting parameter optimization: {len(all_combinations)} combinations")
# Determine number of workers using SystemUtils
if max_workers is None:
max_workers = self.system_utils.get_optimal_workers()
else:
max_workers = min(max_workers, len(all_combinations))
self._log_action("parameter_optimization_started", {
"strategy_class": strategy_class.__name__,
"total_combinations": len(all_combinations),
"max_workers": max_workers,
"strategy_param_grid": param_grid,
"trader_param_grid": trader_param_grid or {}
})
# Run optimization
if max_workers == 1 or len(all_combinations) == 1:
# Single-threaded execution
results = []
for strategy_params, trader_params in all_combinations:
result = self._run_single_combination(strategy_class, strategy_params, trader_params)
results.append(result)
else:
# Multi-threaded execution
results = self._run_parallel_optimization(
strategy_class, all_combinations, max_workers
)
# Sort results by profit ratio
valid_results = [r for r in results if r.get("success", True)]
valid_results.sort(key=lambda x: x.get("profit_ratio", -float('inf')), reverse=True)
logger.info(f"Parameter optimization completed: {len(valid_results)} successful runs")
self._log_action("parameter_optimization_completed", {
"total_runs": len(results),
"successful_runs": len(valid_results),
"failed_runs": len(results) - len(valid_results),
"best_profit_ratio": valid_results[0]["profit_ratio"] if valid_results else None,
"worst_profit_ratio": valid_results[-1]["profit_ratio"] if valid_results else None
})
return results
def _generate_param_combinations(self, param_grid: Dict[str, List]) -> List[Dict]:
"""Generate all parameter combinations from grid."""
if not param_grid:
return [{}]
keys = list(param_grid.keys())
values = list(param_grid.values())
combinations = []
for combination in product(*values):
param_dict = dict(zip(keys, combination))
combinations.append(param_dict)
return combinations
def _run_single_combination(self, strategy_class: type, strategy_params: Dict,
trader_params: Dict) -> Dict[str, Any]:
"""Run backtest for a single parameter combination."""
try:
# Create strategy instance
strategy = strategy_class(params=strategy_params)
# Run backtest
result = self.run_single_strategy(strategy, trader_params)
result["success"] = True
return result
except Exception as e:
logger.error(f"Error in parameter combination {strategy_params}, {trader_params}: {e}")
return {
"strategy_params": strategy_params,
"trader_params": trader_params,
"error": str(e),
"success": False
}
def _run_parallel_optimization(self, strategy_class: type, combinations: List,
max_workers: int) -> List[Dict[str, Any]]:
"""Run parameter optimization in parallel."""
results = []
# Prepare arguments for worker function
worker_args = []
for strategy_params, trader_params in combinations:
args = (strategy_class, strategy_params, trader_params, self.config, self.config.data_file)
worker_args.append(args)
# Execute in parallel
with ProcessPoolExecutor(max_workers=max_workers) as executor:
# Submit all jobs
future_to_params = {
executor.submit(_worker_function, args): args[1:3] # strategy_params, trader_params
for args in worker_args
}
# Collect results as they complete
for future in as_completed(future_to_params):
combo = future_to_params[future]
try:
result = future.result()
results.append(result)
if result.get("success", True):
logger.info(f"Completed: {combo[0]} -> "
f"${result.get('final_usd', 0):.2f} "
f"({result.get('profit_ratio', 0)*100:.2f}%)")
except Exception as e:
logger.error(f"Worker error for {combo}: {e}")
results.append({
"strategy_params": combo[0],
"trader_params": combo[1],
"error": str(e),
"success": False
})
return results
def get_summary_statistics(self, results: List[Dict[str, Any]]) -> Dict[str, Any]:
"""
Calculate summary statistics across multiple backtest results.
Args:
results: List of backtest results
Returns:
Dict containing summary statistics
"""
valid_results = [r for r in results if r.get("success", True)]
if not valid_results:
return {
"total_runs": len(results),
"successful_runs": 0,
"failed_runs": len(results),
"error": "No valid results to summarize"
}
# Extract metrics
profit_ratios = [r["profit_ratio"] for r in valid_results]
final_balances = [r["final_usd"] for r in valid_results]
n_trades_list = [r["n_trades"] for r in valid_results]
win_rates = [r["win_rate"] for r in valid_results]
max_drawdowns = [r["max_drawdown"] for r in valid_results]
summary = {
"total_runs": len(results),
"successful_runs": len(valid_results),
"failed_runs": len(results) - len(valid_results),
# Profit statistics
"profit_ratio": {
"mean": np.mean(profit_ratios),
"std": np.std(profit_ratios),
"min": np.min(profit_ratios),
"max": np.max(profit_ratios),
"median": np.median(profit_ratios)
},
# Balance statistics
"final_usd": {
"mean": np.mean(final_balances),
"std": np.std(final_balances),
"min": np.min(final_balances),
"max": np.max(final_balances),
"median": np.median(final_balances)
},
# Trading statistics
"n_trades": {
"mean": np.mean(n_trades_list),
"std": np.std(n_trades_list),
"min": np.min(n_trades_list),
"max": np.max(n_trades_list),
"median": np.median(n_trades_list)
},
# Performance statistics
"win_rate": {
"mean": np.mean(win_rates),
"std": np.std(win_rates),
"min": np.min(win_rates),
"max": np.max(win_rates),
"median": np.median(win_rates)
},
"max_drawdown": {
"mean": np.mean(max_drawdowns),
"std": np.std(max_drawdowns),
"min": np.min(max_drawdowns),
"max": np.max(max_drawdowns),
"median": np.median(max_drawdowns)
},
# Best performing run
"best_run": max(valid_results, key=lambda x: x["profit_ratio"]),
"worst_run": min(valid_results, key=lambda x: x["profit_ratio"])
}
return summary
def save_comprehensive_results(self, results: List[Dict[str, Any]],
base_filename: str,
summary: Optional[Dict[str, Any]] = None) -> None:
"""
Save comprehensive backtest results including summary, individual results, and action log.
Args:
results: List of backtest results
base_filename: Base filename (without extension)
summary: Optional summary statistics
"""
try:
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
# 1. Save summary report
if summary is None:
summary = self.get_summary_statistics(results)
summary_data = {
"session_info": {
"timestamp": timestamp,
"session_start": self.session_start_time.isoformat(),
"session_duration_seconds": (datetime.now() - self.session_start_time).total_seconds(),
"config": self.config.__dict__
},
"summary_statistics": summary,
"action_log_summary": {
"total_actions": len(self.action_log),
"action_types": list(set(action["action_type"] for action in self.action_log))
}
}
summary_filename = f"{base_filename}_summary_{timestamp}.json"
with open(f"results/{summary_filename}", 'w') as f:
json.dump(summary_data, f, indent=2, default=str)
logger.info(f"Summary saved to results/{summary_filename}")
# 2. Save detailed results CSV
self.save_results(results, f"{base_filename}_detailed_{timestamp}.csv")
# 3. Save individual strategy results
valid_results = [r for r in results if r.get("success", True)]
for i, result in enumerate(valid_results):
strategy_filename = f"{base_filename}_strategy_{i+1}_{result['strategy_name']}_{timestamp}.json"
# Include trades and detailed info
strategy_data = {
"strategy_info": {
"name": result['strategy_name'],
"params": result.get('strategy_params', {}),
"trader_params": result.get('trader_params', {})
},
"performance": {
"initial_usd": result['initial_usd'],
"final_usd": result['final_usd'],
"profit_ratio": result['profit_ratio'],
"n_trades": result['n_trades'],
"win_rate": result['win_rate'],
"max_drawdown": result['max_drawdown'],
"avg_trade": result['avg_trade'],
"total_fees_usd": result['total_fees_usd']
},
"execution": {
"backtest_duration_seconds": result.get('backtest_duration_seconds', 0),
"data_points_processed": result.get('data_points_processed', 0),
"warmup_complete": result.get('warmup_complete', False)
},
"trades": result.get('trades', [])
}
with open(f"results/{strategy_filename}", 'w') as f:
json.dump(strategy_data, f, indent=2, default=str)
logger.info(f"Strategy {i+1} details saved to results/{strategy_filename}")
# 4. Save complete action log
action_log_filename = f"{base_filename}_actions_{timestamp}.json"
action_log_data = {
"session_info": {
"timestamp": timestamp,
"session_start": self.session_start_time.isoformat(),
"total_actions": len(self.action_log)
},
"actions": self.action_log
}
with open(f"results/{action_log_filename}", 'w') as f:
json.dump(action_log_data, f, indent=2, default=str)
logger.info(f"Action log saved to results/{action_log_filename}")
# 5. Create a master index file
index_filename = f"{base_filename}_index_{timestamp}.json"
index_data = {
"session_info": {
"timestamp": timestamp,
"base_filename": base_filename,
"total_strategies": len(valid_results),
"session_duration_seconds": (datetime.now() - self.session_start_time).total_seconds()
},
"files": {
"summary": summary_filename,
"detailed_csv": f"{base_filename}_detailed_{timestamp}.csv",
"action_log": action_log_filename,
"individual_strategies": [
f"{base_filename}_strategy_{i+1}_{result['strategy_name']}_{timestamp}.json"
for i, result in enumerate(valid_results)
]
},
"quick_stats": {
"best_profit": summary.get("profit_ratio", {}).get("max", 0) if summary.get("profit_ratio") else 0,
"worst_profit": summary.get("profit_ratio", {}).get("min", 0) if summary.get("profit_ratio") else 0,
"avg_profit": summary.get("profit_ratio", {}).get("mean", 0) if summary.get("profit_ratio") else 0,
"total_successful_runs": summary.get("successful_runs", 0),
"total_failed_runs": summary.get("failed_runs", 0)
}
}
with open(f"results/{index_filename}", 'w') as f:
json.dump(index_data, f, indent=2, default=str)
logger.info(f"Master index saved to results/{index_filename}")
print(f"\n📊 Comprehensive results saved:")
print(f" 📋 Summary: results/{summary_filename}")
print(f" 📈 Detailed CSV: results/{base_filename}_detailed_{timestamp}.csv")
print(f" 📝 Action Log: results/{action_log_filename}")
print(f" 📁 Individual Strategies: {len(valid_results)} files")
print(f" 🗂️ Master Index: results/{index_filename}")
except Exception as e:
logger.error(f"Error saving comprehensive results: {e}")
raise
def save_results(self, results: List[Dict[str, Any]], filename: str) -> None:
"""
Save backtest results to file.
Args:
results: List of backtest results
filename: Output filename
"""
try:
# Convert results to DataFrame for easy saving
df_data = []
for result in results:
if result.get("success", True):
row = {
"strategy_name": result.get("strategy_name", ""),
"profit_ratio": result.get("profit_ratio", 0),
"final_usd": result.get("final_usd", 0),
"n_trades": result.get("n_trades", 0),
"win_rate": result.get("win_rate", 0),
"max_drawdown": result.get("max_drawdown", 0),
"avg_trade": result.get("avg_trade", 0),
"total_fees_usd": result.get("total_fees_usd", 0),
"backtest_duration_seconds": result.get("backtest_duration_seconds", 0),
"data_points_processed": result.get("data_points_processed", 0)
}
# Add strategy parameters
strategy_params = result.get("strategy_params", {})
for key, value in strategy_params.items():
row[f"strategy_{key}"] = value
# Add trader parameters
trader_params = result.get("trader_params", {})
for key, value in trader_params.items():
row[f"trader_{key}"] = value
df_data.append(row)
# Save to CSV
df = pd.DataFrame(df_data)
self.storage.save_data(df, filename)
logger.info(f"Results saved to {filename}: {len(df_data)} rows")
except Exception as e:
logger.error(f"Error saving results to {filename}: {e}")
raise
def __repr__(self) -> str:
"""String representation of the backtester."""
return (f"IncBacktester(data_file={self.config.data_file}, "
f"date_range={self.config.start_date} to {self.config.end_date}, "
f"initial_usd=${self.config.initial_usd})")

344
cycles/IncStrategies/inc_trader.py Normal file
View File

@@ -0,0 +1,344 @@
"""
Incremental Trader for backtesting incremental strategies.
This module provides the IncTrader class that manages a single incremental strategy
during backtesting, handling position state, trade execution, and performance tracking.
"""
import pandas as pd
import numpy as np
from typing import Dict, Optional, List, Any
import logging
from dataclasses import dataclass
from .base import IncStrategyBase, IncStrategySignal
from ..market_fees import MarketFees
logger = logging.getLogger(__name__)
@dataclass
class TradeRecord:
"""Record of a completed trade."""
entry_time: pd.Timestamp
exit_time: pd.Timestamp
entry_price: float
exit_price: float
entry_fee: float
exit_fee: float
profit_pct: float
exit_reason: str
strategy_name: str
class IncTrader:
"""
Incremental trader that manages a single strategy during backtesting.
This class handles:
- Strategy initialization and data feeding
- Position management (USD/coin balance)
- Trade execution based on strategy signals
- Performance tracking and metrics collection
- Fee calculation and trade logging
The trader processes data points sequentially, feeding them to the strategy
and executing trades based on the generated signals.
Example:
strategy = IncRandomStrategy(params={"timeframe": "15min"})
trader = IncTrader(
strategy=strategy,
initial_usd=10000,
params={"stop_loss_pct": 0.02}
)
# Process data sequentially
for timestamp, ohlcv_data in data_stream:
trader.process_data_point(timestamp, ohlcv_data)
# Get results
results = trader.get_results()
"""
def __init__(self, strategy: IncStrategyBase, initial_usd: float = 10000,
params: Optional[Dict] = None):
"""
Initialize the incremental trader.
Args:
strategy: Incremental strategy instance
initial_usd: Initial USD balance
params: Trader parameters (stop_loss_pct, take_profit_pct, etc.)
"""
self.strategy = strategy
self.initial_usd = initial_usd
self.params = params or {}
# Position state
self.usd = initial_usd
self.coin = 0.0
self.position = 0 # 0 = no position, 1 = long position
self.entry_price = 0.0
self.entry_time = None
# Performance tracking
self.max_balance = initial_usd
self.drawdowns = []
self.trade_records = []
self.current_timestamp = None
self.current_price = None
# Strategy state
self.data_points_processed = 0
self.warmup_complete = False
# Parameters
self.stop_loss_pct = self.params.get("stop_loss_pct", 0.0)
self.take_profit_pct = self.params.get("take_profit_pct", 0.0)
logger.info(f"IncTrader initialized: strategy={strategy.name}, "
f"initial_usd=${initial_usd}, stop_loss={self.stop_loss_pct*100:.1f}%")
def process_data_point(self, timestamp: pd.Timestamp, ohlcv_data: Dict[str, float]) -> None:
"""
Process a single data point through the strategy and handle trading logic.
Args:
timestamp: Data point timestamp
ohlcv_data: OHLCV data dictionary with keys: open, high, low, close, volume
"""
self.current_timestamp = timestamp
self.current_price = ohlcv_data['close']
self.data_points_processed += 1
try:
# Feed data to strategy (handles timeframe aggregation internally)
result = self.strategy.update_minute_data(timestamp, ohlcv_data)
# Check if strategy is warmed up
if not self.warmup_complete and self.strategy.is_warmed_up:
self.warmup_complete = True
logger.info(f"Strategy {self.strategy.name} warmed up after "
f"{self.data_points_processed} data points")
# Only process signals if strategy is warmed up and we have a complete timeframe bar
if self.warmup_complete and result is not None:
self._process_trading_logic()
# Update performance tracking
self._update_performance_metrics()
except Exception as e:
logger.error(f"Error processing data point at {timestamp}: {e}")
raise
def _process_trading_logic(self) -> None:
"""Process trading logic based on current position and strategy signals."""
if self.position == 0:
# No position - check for entry signals
self._check_entry_signals()
else:
# In position - check for exit signals
self._check_exit_signals()
def _check_entry_signals(self) -> None:
"""Check for entry signals when not in position."""
try:
entry_signal = self.strategy.get_entry_signal()
if entry_signal.signal_type == "ENTRY" and entry_signal.confidence > 0:
self._execute_entry(entry_signal)
except Exception as e:
logger.error(f"Error checking entry signals: {e}")
def _check_exit_signals(self) -> None:
"""Check for exit signals when in position."""
try:
# Check strategy exit signals
exit_signal = self.strategy.get_exit_signal()
if exit_signal.signal_type == "EXIT" and exit_signal.confidence > 0:
exit_reason = exit_signal.metadata.get("type", "STRATEGY_EXIT")
self._execute_exit(exit_reason, exit_signal.price)
return
# Check stop loss
if self.stop_loss_pct > 0:
stop_loss_price = self.entry_price * (1 - self.stop_loss_pct)
if self.current_price <= stop_loss_price:
self._execute_exit("STOP_LOSS", self.current_price)
return
# Check take profit
if self.take_profit_pct > 0:
take_profit_price = self.entry_price * (1 + self.take_profit_pct)
if self.current_price >= take_profit_price:
self._execute_exit("TAKE_PROFIT", self.current_price)
return
except Exception as e:
logger.error(f"Error checking exit signals: {e}")
def _execute_entry(self, signal: IncStrategySignal) -> None:
"""Execute entry trade."""
entry_price = signal.price if signal.price else self.current_price
entry_fee = MarketFees.calculate_okx_taker_maker_fee(self.usd, is_maker=False)
usd_after_fee = self.usd - entry_fee
self.coin = usd_after_fee / entry_price
self.entry_price = entry_price
self.entry_time = self.current_timestamp
self.usd = 0.0
self.position = 1
logger.info(f"ENTRY: {self.strategy.name} at ${entry_price:.2f}, "
f"confidence={signal.confidence:.2f}, fee=${entry_fee:.2f}")
def _execute_exit(self, exit_reason: str, exit_price: Optional[float] = None) -> None:
"""Execute exit trade."""
exit_price = exit_price if exit_price else self.current_price
usd_gross = self.coin * exit_price
exit_fee = MarketFees.calculate_okx_taker_maker_fee(usd_gross, is_maker=False)
self.usd = usd_gross - exit_fee
# Calculate profit
profit_pct = (exit_price - self.entry_price) / self.entry_price
# Record trade
trade_record = TradeRecord(
entry_time=self.entry_time,
exit_time=self.current_timestamp,
entry_price=self.entry_price,
exit_price=exit_price,
entry_fee=MarketFees.calculate_okx_taker_maker_fee(
self.coin * self.entry_price, is_maker=False
),
exit_fee=exit_fee,
profit_pct=profit_pct,
exit_reason=exit_reason,
strategy_name=self.strategy.name
)
self.trade_records.append(trade_record)
# Reset position
self.coin = 0.0
self.position = 0
self.entry_price = 0.0
self.entry_time = None
logger.info(f"EXIT: {self.strategy.name} at ${exit_price:.2f}, "
f"reason={exit_reason}, profit={profit_pct*100:.2f}%, fee=${exit_fee:.2f}")
def _update_performance_metrics(self) -> None:
"""Update performance tracking metrics."""
# Calculate current balance
if self.position == 0:
current_balance = self.usd
else:
current_balance = self.coin * self.current_price
# Update max balance and drawdown
if current_balance > self.max_balance:
self.max_balance = current_balance
drawdown = (self.max_balance - current_balance) / self.max_balance
self.drawdowns.append(drawdown)
def finalize(self) -> None:
"""Finalize trading session (close any open positions)."""
if self.position == 1:
self._execute_exit("EOD", self.current_price)
logger.info(f"Closed final position for {self.strategy.name} at EOD")
def get_results(self) -> Dict[str, Any]:
"""
Get comprehensive trading results.
Returns:
Dict containing performance metrics, trade records, and statistics
"""
final_balance = self.usd
n_trades = len(self.trade_records)
# Calculate statistics
if n_trades > 0:
profits = [trade.profit_pct for trade in self.trade_records]
wins = [p for p in profits if p > 0]
win_rate = len(wins) / n_trades
avg_trade = np.mean(profits)
total_fees = sum(trade.entry_fee + trade.exit_fee for trade in self.trade_records)
else:
win_rate = 0.0
avg_trade = 0.0
total_fees = 0.0
max_drawdown = max(self.drawdowns) if self.drawdowns else 0.0
profit_ratio = (final_balance - self.initial_usd) / self.initial_usd
# Convert trade records to dictionaries
trades = []
for trade in self.trade_records:
trades.append({
'entry_time': trade.entry_time,
'exit_time': trade.exit_time,
'entry': trade.entry_price,
'exit': trade.exit_price,
'profit_pct': trade.profit_pct,
'type': trade.exit_reason,
'fee_usd': trade.entry_fee + trade.exit_fee,
'strategy': trade.strategy_name
})
results = {
"strategy_name": self.strategy.name,
"strategy_params": self.strategy.params,
"trader_params": self.params,
"initial_usd": self.initial_usd,
"final_usd": final_balance,
"profit_ratio": profit_ratio,
"n_trades": n_trades,
"win_rate": win_rate,
"max_drawdown": max_drawdown,
"avg_trade": avg_trade,
"total_fees_usd": total_fees,
"data_points_processed": self.data_points_processed,
"warmup_complete": self.warmup_complete,
"trades": trades
}
# Add first and last trade info if available
if n_trades > 0:
results["first_trade"] = {
"entry_time": self.trade_records[0].entry_time,
"entry": self.trade_records[0].entry_price
}
results["last_trade"] = {
"exit_time": self.trade_records[-1].exit_time,
"exit": self.trade_records[-1].exit_price
}
return results
def get_current_state(self) -> Dict[str, Any]:
"""Get current trader state for debugging."""
return {
"strategy": self.strategy.name,
"position": self.position,
"usd": self.usd,
"coin": self.coin,
"current_price": self.current_price,
"entry_price": self.entry_price,
"data_points_processed": self.data_points_processed,
"warmup_complete": self.warmup_complete,
"n_trades": len(self.trade_records),
"strategy_state": self.strategy.get_current_state_summary()
}
def __repr__(self) -> str:
"""String representation of the trader."""
return (f"IncTrader(strategy={self.strategy.name}, "
f"position={self.position}, usd=${self.usd:.2f}, "
f"trades={len(self.trade_records)})")

5
cycles/IncStrategies/metatrend_strategy.py
View File

@@ -320,12 +320,13 @@ class IncMetaTrendStrategy(IncStrategyBase):
"""
Check if meta-trend exit condition is met.
Exit condition: meta-trend changes from != -1 to == -1
Exit condition: meta-trend changes from != 1 to == -1
(Modified to match original strategy behavior)
Returns:
bool: True if exit condition is met
"""
return (self.previous_meta_trend != -1 and
return (self.previous_meta_trend != 1 and
self.current_meta_trend == -1)
def get_current_state_summary(self) -> Dict[str, Any]: