2025-05-30 16:28:58 +08:00
# Simplified Crypto Trading Bot Platform: Product Requirements Document (PRD)
**Version:** 1.0
**Date:** May 30, 2025
**Author:** Vasily
**Status:** Draft
2025-06-06 20:33:29 +08:00
> **Note on Implementation Status:** This document describes the complete vision for the platform. As of the current development phase, many components like the **Strategy Engine**, **Bot Manager**, and **Backtesting Engine** are planned but not yet implemented. For a detailed view of the current status, please refer to the main `CONTEXT.md` file.
2025-05-30 16:28:58 +08:00
## Executive Summary
This PRD outlines the development of a simplified crypto trading bot platform that enables strategy testing, development, and execution without the complexity of microservices and advanced monitoring. The goal is to create a functional system within 1-2 weeks that allows for strategy testing while establishing a foundation that can scale in the future. The platform addresses key requirements including data collection, strategy execution, visualization, and backtesting capabilities in a monolithic architecture optimized for internal use.
2025-06-06 20:33:29 +08:00
---
*Back to [Main Documentation (`../README.md` )]*
2025-05-30 16:28:58 +08:00
## Current Requirements & Constraints
- **Speed to Deployment**: System must be functional within 1-2 weeks
- **Scale**: Support for 5-10 concurrent trading bots
- **Architecture**: Monolithic application instead of microservices
- **User Access**: Internal use only initially (no multi-user authentication)
- **Infrastructure**: Simplified deployment without Kubernetes/Docker Swarm
- **Monitoring**: Basic logging for modules
## System Architecture
### High-Level Architecture
The platform will follow a monolithic architecture pattern to enable rapid development while providing clear separation between components:
### Data Flow Architecture
```
OKX Exchange API (WebSocket)
↓
Data Collector → OHLCV Aggregator → PostgreSQL (market_data)
↓ ↓
[Optional] Raw Trade Storage Redis Pub/Sub → Strategy Engine (JSON configs)
↓ ↓
Files/Database (raw_trades) Signal Generation → Bot Manager
↓
PostgreSQL (signals, trades, bot_performance)
↓
Dashboard (REST API) ← PostgreSQL (historical data)
↑
Real-time Updates ← Redis Channels
```
**Data Processing Priority**:
1. **Real-time** : Raw data → OHLCV candles → Redis → Bots (primary flow)
2. **Historical** : OHLCV data from PostgreSQL for backtesting and charts
3. **Advanced Analysis** : Raw trade data (if stored) for detailed backtesting
### Redis Channel Design
```python
# Real-time market data distribution
MARKET_DATA_CHANNEL = "market:{symbol}" # OHLCV updates
BOT_SIGNALS_CHANNEL = "signals:{bot_id}" # Trading decisions
BOT_STATUS_CHANNEL = "status:{bot_id}" # Bot lifecycle events
SYSTEM_EVENTS_CHANNEL = "system:events" # Global notifications
```
### Configuration Strategy
**PostgreSQL for**: Market data, bot instances, trades, signals, performance metrics
**JSON files for**: Strategy parameters, bot configurations (rapid testing and parameter tuning)
```json
// config/strategies/ema_crossover.json
{
"strategy_name": "EMA_Crossover",
"parameters": {
"fast_period": 12,
"slow_period": 26,
"risk_percentage": 0.02
}
}
// config/bots/bot_001.json
{
"bot_id": "bot_001",
"strategy_file": "ema_crossover.json",
"symbol": "BTC-USDT",
"virtual_balance": 10000,
"enabled": true
}
```
### Error Handling Strategy
**Bot Crash Recovery**:
- Monitor bot processes every 30 seconds
- Auto-restart crashed bots if status = 'active'
- Log all crashes with stack traces
- Maximum 3 restart attempts per hour
**Exchange Connection Issues**:
- Retry with exponential backoff (1s, 2s, 4s, 8s, max 60s)
- Switch to backup WebSocket connection if available
- Log connection quality metrics
**Database Errors**:
- Continue operation with in-memory cache for up to 5 minutes
- Queue operations for retry when connection restored
- Alert on prolonged database disconnection
**Application Restart Recovery**:
- Read bot states from database on startup
- Restore active bots to 'active' status
- Resume data collection for all monitored symbols
### Component Details and Functional Requirements
1. **Data Collection Module**
- Connect to exchange APIs (OKX initially) via WebSocket
- Aggregate real-time trades into OHLCV candles (1m, 5m, 15m, 1h, 4h, 1d)
- Store OHLCV data in PostgreSQL for bot operations and backtesting
- Send real-time candle updates through Redis
- Optional: Store raw trade data for advanced backtesting
**FR-001: Unified Data Provider Interface**
- Support multiple exchanges through standardized adapters
- Real-time OHLCV aggregation with WebSocket connections
- Primary focus on candle data, raw data storage optional
- Data validation and error handling mechanisms
**FR-002: Market Data Processing**
- OHLCV aggregation with configurable timeframes (1m base, higher timeframes derived)
- Technical indicator calculation (SMA, EMA, RSI, MACD, Bollinger Bands) on OHLCV data
- Data normalization across different exchanges
- Time alignment following exchange standards (right-aligned candles)
2. **Strategy Engine**
- Provide unified interface for all trading strategies
- Support multiple strategy types with common parameter structure
- Generate trading signals based on market data
- Log strategy performance and signals
- Strategy implementation as a class.
**FR-003: Strategy Framework**
- Base strategy class with standardized interface
- Support for multiple strategy types
- Parameter configuration and optimization tools (JSON for the parameters)
- Signal generation with confidence scoring
**FR-004: Signal Processing**
- Real-time signal calculation and validation
- Signal persistence for analysis and debugging
- Multi-timeframe analysis capabilities
- Custom indicator development support
3. **Bot Manager**
- Create and manage up to 10 concurrent trading bots
- Configure bot parameters and associated strategies
- Start/stop individual bots
- Track bot status and performance
**FR-005: Bot Lifecycle Management**
- Bot creation with strategy and parameter selection
- Start/stop/pause functionality with state persistence
- Configuration management
- Resource allocation and monitoring (in future)
**FR-006: Portfolio Management**
- Position tracking and balance management
- Risk management controls (stop-loss, take-profit, position sizing)
- Multi-bot coordination and conflict resolution (in future)
- Real-time portfolio valuation (in future)
5. **Trading Execution**
- Simulate or execute trades based on configuration
- Stores trade information in database
**FR-007: Order Management**
- Order placement with multiple order types (market, limit, stop)
- Order tracking and status monitoring (in future)
- Execution confirmation and reconciliation (in future)
- Fee calculation and tracking (in future)
**FR-008: Risk Controls**
- Pre-trade risk validation
- Position limits and exposure controls (in future)
- Emergency stop mechanisms (in future)
- Compliance monitoring and reporting (in future)
4. **Database (PostgreSQL)**
- Store market data, bot configurations, and trading history
- Optimized schema for time-series data without complexity
- Support for data querying and aggregation
**Database (JSON)**
- Store strategy parameters and bot onfiguration in JSON in the beginning for simplicity of editing and testing
5. **Backtesting Engine**
- Run simulations on historical data using vectorized operations for speed
- Calculate performance metrics
- Support multiple timeframes and strategy parameter testing
- Generate comparison reports between strategies
**FR-009: Historical Simulation**
- Strategy backtesting on historical market data
- Performance metric calculation (Sharpe ratio, drawdown, win rate, total return)
- Parameter optimization through grid search (limited combinations for speed) (in future)
- Side-by-side strategy comparison with statistical significance
**FR-010: Simulation Engine**
- Vectorized signal calculation using pandas operations
- Realistic fee modeling (0.1% per trade for OKX)
- Look-ahead bias prevention with proper timestamp handling
- Configurable test periods (1 day to 24 months)
6. **Dashboard & Visualization**
- Display real-time market data and bot status
- Show portfolio value progression over time
- Visualize trade history with buy/sell markers on price charts
- Provide simple bot control interface (start/stop/configure)
**FR-011: Dashboard Interface**
- Real-time bot monitoring with status indicators
- Portfolio performance charts (total value, cash vs crypto allocation)
- Trade history table with P& L per trade
- Simple bot configuration forms for JSON parameter editing
**FR-012: Data Visualization**
- Interactive price charts with strategy signal overlays
- Portfolio value progression charts
- Performance comparison tables (multiple bots side-by-side)
- Fee tracking and total cost analysis
### Non-Functional Requirements
1 Performance Requirements
**NFR-001: Latency**
- Market data processing: < 100ms from exchange to database
- Signal generation: < 500ms for standard strategies
- API response time: < 200ms for 95 % of requests
- Dashboard updates: < 2 seconds for real-time data
**NFR-002: Scalability**
- Database queries scalable to 1M+ records per table
- Horizontal scaling capability for all services (in future)
2. Reliability Requirements
**NFR-003: Availability**
- System uptime: 99.5% excluding planned maintenance
- Data collection: 99.9% uptime during market hours
- Automatic failover for critical services
- Graceful degradation during partial outages
**NFR-004: Data Integrity**
- Zero data loss for executed trades
- Transactional consistency for all financial operations
- Regular database backups with point-in-time recovery
- Data validation and error correction mechanisms
3. Security Requirements
**NFR-005: Authentication & Authorization** (in future)
**NFR-006: Data Protection**
- End-to-end encryption for sensitive data (in future)
- Secure storage of API keys and credentials
- Regular security audits and penetration testing (in future)
- Compliance with financial data protection regulations (in future)
## Technical Implementation
### Database Schema
The database schema separates frequently-accessed OHLCV data from raw tick data to optimize performance and storage.
```sql
-- OHLCV Market Data (primary table for bot operations)
CREATE TABLE market_data (
id SERIAL PRIMARY KEY,
exchange VARCHAR(50) NOT NULL DEFAULT 'okx',
symbol VARCHAR(20) NOT NULL,
timeframe VARCHAR(5) NOT NULL, -- 1m, 5m, 15m, 1h, 4h, 1d
timestamp TIMESTAMPTZ NOT NULL,
open DECIMAL(18,8) NOT NULL,
high DECIMAL(18,8) NOT NULL,
low DECIMAL(18,8) NOT NULL,
close DECIMAL(18,8) NOT NULL,
volume DECIMAL(18,8) NOT NULL,
trades_count INTEGER, -- number of trades in this candle
created_at TIMESTAMPTZ DEFAULT NOW(),
UNIQUE(exchange, symbol, timeframe, timestamp)
);
CREATE INDEX idx_market_data_lookup ON market_data(symbol, timeframe, timestamp);
CREATE INDEX idx_market_data_recent ON market_data(timestamp DESC) WHERE timestamp > NOW() - INTERVAL '7 days';
-- Raw Trade Data (optional, for detailed backtesting only)
CREATE TABLE raw_trades (
id SERIAL PRIMARY KEY,
exchange VARCHAR(50) NOT NULL DEFAULT 'okx',
symbol VARCHAR(20) NOT NULL,
timestamp TIMESTAMPTZ NOT NULL,
type VARCHAR(10) NOT NULL, -- trade, order, balance, tick, books
data JSONB NOT NULL, -- response from the exchange
created_at TIMESTAMPTZ DEFAULT NOW()
) PARTITION BY RANGE (timestamp);
CREATE INDEX idx_raw_trades_symbol_time ON raw_trades(symbol, timestamp);
-- Monthly partitions for raw data (if using raw data)
-- CREATE TABLE raw_trades_y2024m01 PARTITION OF raw_trades
-- FOR VALUES FROM ('2024-01-01') TO ('2024-02-01');
-- Bot Management (simplified)
CREATE TABLE bots (
id SERIAL PRIMARY KEY,
name VARCHAR(100) NOT NULL,
strategy_name VARCHAR(50) NOT NULL,
symbol VARCHAR(20) NOT NULL,
timeframe VARCHAR(5) NOT NULL,
status VARCHAR(20) NOT NULL DEFAULT 'inactive', -- active, inactive, error
config_file VARCHAR(200), -- path to JSON config
virtual_balance DECIMAL(18,8) DEFAULT 10000,
current_balance DECIMAL(18,8) DEFAULT 10000,
last_heartbeat TIMESTAMPTZ,
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW()
);
-- Trading Signals (for analysis and debugging)
CREATE TABLE signals (
id SERIAL PRIMARY KEY,
bot_id INTEGER REFERENCES bots(id),
timestamp TIMESTAMPTZ NOT NULL,
signal_type VARCHAR(10) NOT NULL, -- buy, sell, hold
price DECIMAL(18,8),
confidence DECIMAL(5,4),
indicators JSONB, -- technical indicator values
created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE INDEX idx_signals_bot_time ON signals(bot_id, timestamp);
-- Trade Execution Records
CREATE TABLE trades (
id SERIAL PRIMARY KEY,
bot_id INTEGER REFERENCES bots(id),
signal_id INTEGER REFERENCES signals(id),
timestamp TIMESTAMPTZ NOT NULL,
side VARCHAR(5) NOT NULL, -- buy, sell
price DECIMAL(18,8) NOT NULL,
quantity DECIMAL(18,8) NOT NULL,
fees DECIMAL(18,8) DEFAULT 0,
pnl DECIMAL(18,8), -- profit/loss for this trade
balance_after DECIMAL(18,8), -- portfolio balance after trade
created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE INDEX idx_trades_bot_time ON trades(bot_id, timestamp);
-- Performance Snapshots (for plotting portfolio over time)
CREATE TABLE bot_performance (
id SERIAL PRIMARY KEY,
bot_id INTEGER REFERENCES bots(id),
timestamp TIMESTAMPTZ NOT NULL,
total_value DECIMAL(18,8) NOT NULL, -- current portfolio value
cash_balance DECIMAL(18,8) NOT NULL,
crypto_balance DECIMAL(18,8) NOT NULL,
total_trades INTEGER DEFAULT 0,
winning_trades INTEGER DEFAULT 0,
total_fees DECIMAL(18,8) DEFAULT 0,
created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE INDEX idx_bot_performance_bot_time ON bot_performance(bot_id, timestamp);
```
**Data Storage Strategy**:
- **OHLCV Data**: Primary source for bot operations, kept indefinitely, optimized indexes
- **Raw Trade Data**: Optional table, only if detailed backtesting needed, can be partitioned monthly
- **Alternative for Raw Data**: Store in compressed files (Parquet/CSV) instead of database for cost efficiency
**MVP Approach**: Start with OHLCV data only, add raw data storage later if advanced backtesting requires it.
### Technology Stack
The platform will be built using the following technologies:
- **Backend Framework**: Python 3.10+ with Dash (includes built-in Flask server for REST API endpoints)
- **Database**: PostgreSQL 14+ (with TimescaleDB extension for time-series optimization)
- **Real-time Messaging**: Redis (for pub/sub messaging between components)
- **Frontend**: Dash with Plotly (for visualization and control interface) and Mantine UI components
- **Configuration**: JSON files for strategy parameters and bot configurations
- **Deployment**: Docker container setup for development and production
### API Design
**Dash Callbacks**: Real-time updates and user interactions
**REST Endpoints**: Historical data queries for backtesting and analysis
```python
# Built-in Flask routes for historical data
@app .server.route('/api/bot/< bot_id > /trades')
@app .server.route('/api/market/< symbol > /history')
@app .server.route('/api/backtest/results/< test_id > ')
```
### Data Flow
The data flow follows a simple pattern to ensure efficient processing:
1. **Market Data Collection** :
- Collector fetches data from exchange APIs
- Raw data is stored in PostgreSQL
- Processed data (e.g., OHLCV candles) are calculated and stored
- Real-time updates are published to Redis channels
2. **Signal Generation** :
- Bots subscribe to relevant data channels and generate signals based on the strategy
- Signals are stored in database and published to Redis
3. **Trade Execution** :
- Bot manager receives signals from strategies
- Validates signals against bot parameters and portfolio
- Simulates or executes trades based on configuration
- Stores trade information in database
4. **Visualization** :
- Dashboard subscribes to real-time data and trading updates
- Queries historical data for charts and performance metrics
- Provides interface for bot management and configuration
## Development Roadmap
### Phase 1: Foundation (Days 1-5)
**Objective**: Establish core system components and data flow
1. **Day 1-2** : Database Setup and Data Collection
- Set up PostgreSQL with initial schema
- Implement OKX API connector
- Create data storage and processing logic
2. **Day 3-4** : Strategy Engine and Bot Manager
- Develop strategy interface and 1-2 example strategies
- Create bot manager with basic controls
- Implement Redis for real-time messaging
3. **Day 5** : Basic Visualization
- Set up Dash/Plotly for simple charts
- Create basic dashboard layout
- Connect to real-time data sources
- Create mockup strategies and bots
### Phase 2: Core Functionality (Days 6-10)
**Objective**: Complete essential features for strategy testing
1. **Day 6-7** : Backtesting Engine
- Get historical data from the database or file (have for BTC/USDT in csv format)
- Create performance calculation metrics
- Develop strategy comparison tools
2. **Day 8-9** : Trading Logic
- Implement virtual trading capability
- Create trade execution logic
- Develop portfolio tracking
3. **Day 10** : Dashboard Enhancement
- Improve visualization components
- Add bot control interface
- Implement real-time performance monitoring
### Phase 3: Refinement (Days 11-14)
**Objective**: Polish system and prepare for ongoing use
1. **Day 11-12** : Testing and Debugging
- Comprehensive system testing
- Fix identified issues
- Performance optimization
2. **Day 13-14** : Documentation and Deployment
- Create user documentation
- Prepare deployment process
- Set up basic monitoring
## Technical Considerations
### Scalability Path
While the initial system is designed as a monolithic application for rapid development, several considerations ensure future scalability:
1. **Module Separation** : Clear boundaries between components enable future extraction into microservices
2. **Database Design** : Schema supports partitioning and sharding for larger data volumes
3. **Message Queue** : Redis implementation paves way for more robust messaging (Kafka/RabbitMQ)
4. **API-First Design** : Internal components communicate through well-defined interfaces
### Time Aggregation
Special attention is given to time aggregation to ensure consistency with exchanges:
```python
def aggregate_candles(trades, timeframe, alignment='right'):
"""
Aggregate trade data into OHLCV candles with consistent timestamp alignment.
Parameters:
- trades: List of trade dictionaries with timestamp and price
- timeframe: String representing the timeframe (e.g., '1m', '5m', '1h')
- alignment: String indicating timestamp alignment ('right' or 'left')
Returns:
- Dictionary with OHLCV data
"""
# Convert timeframe to pandas offset
if timeframe.endswith('m'):
offset = pd.Timedelta(minutes=int(timeframe[:-1]))
elif timeframe.endswith('h'):
offset = pd.Timedelta(hours=int(timeframe[:-1]))
elif timeframe.endswith('d'):
offset = pd.Timedelta(days=int(timeframe[:-1]))
# Create DataFrame from trades
df = pd.DataFrame(trades)
# Convert timestamps to pandas datetime
df['timestamp'] = pd.to_datetime(df['timestamp'], unit='ms')
# Floor timestamps to timeframe
if alignment == 'right':
df['candle_time'] = df['timestamp'].dt.floor(offset)
else:
df['candle_time'] = df['timestamp'].dt.ceil(offset) - offset
# Aggregate to OHLCV
candles = df.groupby('candle_time').agg({
'price': ['first', 'max', 'min', 'last'],
'amount': 'sum'
}).reset_index()
# Rename columns
candles.columns = ['timestamp', 'open', 'high', 'low', 'close', 'volume']
return candles
```
### Performance Optimization
For the initial release, several performance optimizations are implemented:
1. **Database Indexing** : Proper indexes on timestamp and symbol fields
2. **Query Optimization** : Prepared statements and efficient query patterns
3. **Connection Pooling** : Database connection management to prevent leaks
4. **Data Aggregation** : Pre-calculation of common time intervals
5. **Memory Management** : Proper cleanup of data objects after processing
## User Interface
The initial user interface focuses on functionality over aesthetics, providing essential controls and visualizations, minimalistic design.
1. **Market Data View**
- Real-time price charts for monitored symbols
- Order book visualization
- Recent trades list
2. **Bot Management**
- Create/configure bot interface
- Start/stop controls
- Status indicators
3. **Strategy Dashboard**
- Strategy selection and configuration
- Signal visualization
- Performance metrics
4. **Backtesting Interface**
- Historical data selection
- Strategy parameter configuration
- Results visualization
## Risk Management & Mitigation
### Technical Risks
**Risk:** Exchange API rate limiting affecting data collection
**Mitigation:** Implement intelligent rate limiting, multiple API keys, and fallback data sources
**Risk:** Database performance degradation with large datasets
**Mitigation:** Implement data partitioning, archival strategies, and query optimization (in future)
**Risk:** System downtime during market volatility
**Mitigation:** Design redundant systems, implement circuit breakers, and emergency procedures (in future)
### Business Risks
**Risk:** Regulatory changes affecting crypto trading
**Mitigation:** Implement compliance monitoring, maintain regulatory awareness, design for adaptability
**Risk:** Competition from established trading platforms
**Mitigation:** Focus on unique value propositions, rapid feature development, strong user experience
### 8.3 User Risks
**Risk:** User losses due to platform errors
**Mitigation:** Comprehensive testing, simulation modes, risk warnings, and liability disclaimers
## Future Expansion
While keeping the initial implementation simple, the design accommodates future enhancements:
1. **Authentication System** : Add multi-user support with role-based access
2. **Advanced Strategies** : Support for machine learning and AI-based strategies
3. **Multi-Exchange Support** : Expand beyond OKX to other exchanges
4. **Microservices Migration** : Extract components into separate services
5. **Advanced Monitoring** : Integration with Prometheus/Grafana
6. **Cloud Deployment** : Support for AWS/GCP/Azure deployment
## Success Metrics
The platform's success will be measured by these key metrics:
1. **Development Timeline** : Complete core functionality within 14 days
2. **System Stability** : Maintain 99% uptime during internal testing. System should monitor itself and restart if needed (all or just modules)
3. **Strategy Testing** : Successfully backtest at least 3 different strategies
4. **Bot Performance** : Run at least 2 bots concurrently for 72+ hours
