# Simplified Crypto Trading Bot Platform: Product Requirements Document (PRD)

**Version:** 1.0  
**Date:** May 30, 2025  
**Author:** Vasily
**Status:** Draft  

## Executive Summary

This PRD outlines the development of a simplified crypto trading bot platform that enables strategy testing, development, and execution without the complexity of microservices and advanced monitoring. The goal is to create a functional system within 1-2 weeks that allows for strategy testing while establishing a foundation that can scale in the future. The platform addresses key requirements including data collection, strategy execution, visualization, and backtesting capabilities in a monolithic architecture optimized for internal use.

## Current Requirements & Constraints

- **Speed to Deployment**: System must be functional within 1-2 weeks
- **Scale**: Support for 5-10 concurrent trading bots
- **Architecture**: Monolithic application instead of microservices
- **User Access**: Internal use only initially (no multi-user authentication)
- **Infrastructure**: Simplified deployment without Kubernetes/Docker Swarm
- **Monitoring**: Basic logging for modules

## System Architecture

### High-Level Architecture

The platform will follow a monolithic architecture pattern to enable rapid development while providing clear separation between components:

### Data Flow Architecture

```
OKX Exchange API (WebSocket)
       ↓
Data Collector → OHLCV Aggregator → PostgreSQL (market_data)
       ↓                              ↓
[Optional] Raw Trade Storage     Redis Pub/Sub → Strategy Engine (JSON configs)
       ↓                              ↓
Files/Database (raw_trades)      Signal Generation → Bot Manager
                                       ↓
                              PostgreSQL (signals, trades, bot_performance)
                                       ↓
                              Dashboard (REST API) ← PostgreSQL (historical data)
                                       ↑
                              Real-time Updates ← Redis Channels
```

**Data Processing Priority**:
1. **Real-time**: Raw data → OHLCV candles → Redis → Bots (primary flow)
2. **Historical**: OHLCV data from PostgreSQL for backtesting and charts
3. **Advanced Analysis**: Raw trade data (if stored) for detailed backtesting

### Redis Channel Design

```python
# Real-time market data distribution
MARKET_DATA_CHANNEL = "market:{symbol}"           # OHLCV updates
BOT_SIGNALS_CHANNEL = "signals:{bot_id}"         # Trading decisions
BOT_STATUS_CHANNEL = "status:{bot_id}"           # Bot lifecycle events
SYSTEM_EVENTS_CHANNEL = "system:events"         # Global notifications
```

### Configuration Strategy

**PostgreSQL for**: Market data, bot instances, trades, signals, performance metrics
**JSON files for**: Strategy parameters, bot configurations (rapid testing and parameter tuning)

```json
// config/strategies/ema_crossover.json
{
  "strategy_name": "EMA_Crossover",
  "parameters": {
    "fast_period": 12,
    "slow_period": 26,
    "risk_percentage": 0.02
  }
}

// config/bots/bot_001.json  
{
  "bot_id": "bot_001",
  "strategy_file": "ema_crossover.json",
  "symbol": "BTC-USDT",
  "virtual_balance": 10000,
  "enabled": true
}
```

### Error Handling Strategy

**Bot Crash Recovery**:
- Monitor bot processes every 30 seconds
- Auto-restart crashed bots if status = 'active'
- Log all crashes with stack traces
- Maximum 3 restart attempts per hour

**Exchange Connection Issues**:
- Retry with exponential backoff (1s, 2s, 4s, 8s, max 60s)
- Switch to backup WebSocket connection if available
- Log connection quality metrics

**Database Errors**:
- Continue operation with in-memory cache for up to 5 minutes
- Queue operations for retry when connection restored
- Alert on prolonged database disconnection

**Application Restart Recovery**:
- Read bot states from database on startup
- Restore active bots to 'active' status
- Resume data collection for all monitored symbols

### Component Details and Functional Requirements

1. **Data Collection Module**
   - Connect to exchange APIs (OKX initially) via WebSocket
   - Aggregate real-time trades into OHLCV candles (1m, 5m, 15m, 1h, 4h, 1d)
   - Store OHLCV data in PostgreSQL for bot operations and backtesting
   - Send real-time candle updates through Redis
   - Optional: Store raw trade data for advanced backtesting

   **FR-001: Unified Data Provider Interface**
   - Support multiple exchanges through standardized adapters
   - Real-time OHLCV aggregation with WebSocket connections
   - Primary focus on candle data, raw data storage optional
   - Data validation and error handling mechanisms

   **FR-002: Market Data Processing**
   - OHLCV aggregation with configurable timeframes (1m base, higher timeframes derived)
   - Technical indicator calculation (SMA, EMA, RSI, MACD, Bollinger Bands) on OHLCV data
   - Data normalization across different exchanges
   - Time alignment following exchange standards (right-aligned candles)

2. **Strategy Engine**
   - Provide unified interface for all trading strategies
   - Support multiple strategy types with common parameter structure
   - Generate trading signals based on market data
   - Log strategy performance and signals
   - Strategy implementation as a class.

      **FR-003: Strategy Framework**
      - Base strategy class with standardized interface
      - Support for multiple strategy types
      - Parameter configuration and optimization tools (JSON for the parameters)
      - Signal generation with confidence scoring

      **FR-004: Signal Processing**
      - Real-time signal calculation and validation
      - Signal persistence for analysis and debugging
      - Multi-timeframe analysis capabilities
      - Custom indicator development support

3. **Bot Manager**
   - Create and manage up to 10 concurrent trading bots
   - Configure bot parameters and associated strategies
   - Start/stop individual bots
   - Track bot status and performance

      **FR-005: Bot Lifecycle Management**
      - Bot creation with strategy and parameter selection
      - Start/stop/pause functionality with state persistence
      - Configuration management
      - Resource allocation and monitoring (in future)

      **FR-006: Portfolio Management**
      - Position tracking and balance management
      - Risk management controls (stop-loss, take-profit, position sizing)
      - Multi-bot coordination and conflict resolution (in future)
      - Real-time portfolio valuation (in future)

5. **Trading Execution**
   - Simulate or execute trades based on configuration
   - Stores trade information in database

   **FR-007: Order Management**
   - Order placement with multiple order types (market, limit, stop)
   - Order tracking and status monitoring (in future)
   - Execution confirmation and reconciliation (in future)
   - Fee calculation and tracking (in future)

   **FR-008: Risk Controls**
   - Pre-trade risk validation
   - Position limits and exposure controls (in future)
   - Emergency stop mechanisms (in future)
   - Compliance monitoring and reporting (in future)

4. **Database (PostgreSQL)**
   - Store market data, bot configurations, and trading history
   - Optimized schema for time-series data without complexity
   - Support for data querying and aggregation
   **Database (JSON)**
   - Store strategy parameters and bot onfiguration in JSON in the beginning for simplicity of editing and testing

5. **Backtesting Engine**
   - Run simulations on historical data using vectorized operations for speed
   - Calculate performance metrics
   - Support multiple timeframes and strategy parameter testing
   - Generate comparison reports between strategies

   **FR-009: Historical Simulation**
   - Strategy backtesting on historical market data
   - Performance metric calculation (Sharpe ratio, drawdown, win rate, total return)
   - Parameter optimization through grid search (limited combinations for speed) (in future)
   - Side-by-side strategy comparison with statistical significance

   **FR-010: Simulation Engine**  
   - Vectorized signal calculation using pandas operations
   - Realistic fee modeling (0.1% per trade for OKX)
   - Look-ahead bias prevention with proper timestamp handling
   - Configurable test periods (1 day to 24 months)

6. **Dashboard & Visualization**
   - Display real-time market data and bot status
   - Show portfolio value progression over time
   - Visualize trade history with buy/sell markers on price charts
   - Provide simple bot control interface (start/stop/configure)

   **FR-011: Dashboard Interface**
   - Real-time bot monitoring with status indicators
   - Portfolio performance charts (total value, cash vs crypto allocation)
   - Trade history table with P&L per trade
   - Simple bot configuration forms for JSON parameter editing

   **FR-012: Data Visualization**
   - Interactive price charts with strategy signal overlays
   - Portfolio value progression charts
   - Performance comparison tables (multiple bots side-by-side)
   - Fee tracking and total cost analysis

### Non-Functional Requirements

1 Performance Requirements
**NFR-001: Latency**
- Market data processing: <100ms from exchange to database
- Signal generation: <500ms for standard strategies
- API response time: <200ms for 95% of requests
- Dashboard updates: <2 seconds for real-time data

**NFR-002: Scalability**
- Database queries scalable to 1M+ records per table
- Horizontal scaling capability for all services (in future)

2. Reliability Requirements
**NFR-003: Availability**
- System uptime: 99.5% excluding planned maintenance
- Data collection: 99.9% uptime during market hours
- Automatic failover for critical services
- Graceful degradation during partial outages

**NFR-004: Data Integrity**
- Zero data loss for executed trades
- Transactional consistency for all financial operations
- Regular database backups with point-in-time recovery
- Data validation and error correction mechanisms

3. Security Requirements
**NFR-005: Authentication & Authorization** (in future)

**NFR-006: Data Protection**
- End-to-end encryption for sensitive data (in future)
- Secure storage of API keys and credentials
- Regular security audits and penetration testing (in future)
- Compliance with financial data protection regulations (in future)

## Technical Implementation

### Database Schema

The database schema separates frequently-accessed OHLCV data from raw tick data to optimize performance and storage.

```sql
-- OHLCV Market Data (primary table for bot operations)
CREATE TABLE market_data (
    id SERIAL PRIMARY KEY,
    exchange VARCHAR(50) NOT NULL DEFAULT 'okx',
    symbol VARCHAR(20) NOT NULL,
    timeframe VARCHAR(5) NOT NULL, -- 1m, 5m, 15m, 1h, 4h, 1d
    timestamp TIMESTAMPTZ NOT NULL,
    open DECIMAL(18,8) NOT NULL,
    high DECIMAL(18,8) NOT NULL,
    low DECIMAL(18,8) NOT NULL,
    close DECIMAL(18,8) NOT NULL,
    volume DECIMAL(18,8) NOT NULL,
    trades_count INTEGER, -- number of trades in this candle
    created_at TIMESTAMPTZ DEFAULT NOW(),
    UNIQUE(exchange, symbol, timeframe, timestamp)
);
CREATE INDEX idx_market_data_lookup ON market_data(symbol, timeframe, timestamp);
CREATE INDEX idx_market_data_recent ON market_data(timestamp DESC) WHERE timestamp > NOW() - INTERVAL '7 days';

-- Raw Trade Data (optional, for detailed backtesting only)
CREATE TABLE raw_trades (
    id SERIAL PRIMARY KEY,
    exchange VARCHAR(50) NOT NULL DEFAULT 'okx',
    symbol VARCHAR(20) NOT NULL,
    timestamp TIMESTAMPTZ NOT NULL,
    type VARCHAR(10) NOT NULL, -- trade, order, balance, tick, books
    data JSONB NOT NULL,   -- response from the exchange
    created_at TIMESTAMPTZ DEFAULT NOW()
) PARTITION BY RANGE (timestamp);
CREATE INDEX idx_raw_trades_symbol_time ON raw_trades(symbol, timestamp);

-- Monthly partitions for raw data (if using raw data)
-- CREATE TABLE raw_trades_y2024m01 PARTITION OF raw_trades 
-- FOR VALUES FROM ('2024-01-01') TO ('2024-02-01');

-- Bot Management (simplified)
CREATE TABLE bots (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100) NOT NULL,
    strategy_name VARCHAR(50) NOT NULL,
    symbol VARCHAR(20) NOT NULL,
    timeframe VARCHAR(5) NOT NULL,
    status VARCHAR(20) NOT NULL DEFAULT 'inactive', -- active, inactive, error
    config_file VARCHAR(200), -- path to JSON config
    virtual_balance DECIMAL(18,8) DEFAULT 10000,
    current_balance DECIMAL(18,8) DEFAULT 10000,
    last_heartbeat TIMESTAMPTZ,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

-- Trading Signals (for analysis and debugging)
CREATE TABLE signals (
    id SERIAL PRIMARY KEY,
    bot_id INTEGER REFERENCES bots(id),
    timestamp TIMESTAMPTZ NOT NULL,
    signal_type VARCHAR(10) NOT NULL, -- buy, sell, hold
    price DECIMAL(18,8),
    confidence DECIMAL(5,4),
    indicators JSONB, -- technical indicator values
    created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE INDEX idx_signals_bot_time ON signals(bot_id, timestamp);

-- Trade Execution Records
CREATE TABLE trades (
    id SERIAL PRIMARY KEY,
    bot_id INTEGER REFERENCES bots(id),
    signal_id INTEGER REFERENCES signals(id),
    timestamp TIMESTAMPTZ NOT NULL,
    side VARCHAR(5) NOT NULL, -- buy, sell
    price DECIMAL(18,8) NOT NULL,
    quantity DECIMAL(18,8) NOT NULL,
    fees DECIMAL(18,8) DEFAULT 0,
    pnl DECIMAL(18,8), -- profit/loss for this trade
    balance_after DECIMAL(18,8), -- portfolio balance after trade
    created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE INDEX idx_trades_bot_time ON trades(bot_id, timestamp);

-- Performance Snapshots (for plotting portfolio over time)
CREATE TABLE bot_performance (
    id SERIAL PRIMARY KEY,
    bot_id INTEGER REFERENCES bots(id),
    timestamp TIMESTAMPTZ NOT NULL,
    total_value DECIMAL(18,8) NOT NULL, -- current portfolio value
    cash_balance DECIMAL(18,8) NOT NULL,
    crypto_balance DECIMAL(18,8) NOT NULL,
    total_trades INTEGER DEFAULT 0,
    winning_trades INTEGER DEFAULT 0,
    total_fees DECIMAL(18,8) DEFAULT 0,
    created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE INDEX idx_bot_performance_bot_time ON bot_performance(bot_id, timestamp);
```

**Data Storage Strategy**: 
- **OHLCV Data**: Primary source for bot operations, kept indefinitely, optimized indexes
- **Raw Trade Data**: Optional table, only if detailed backtesting needed, can be partitioned monthly
- **Alternative for Raw Data**: Store in compressed files (Parquet/CSV) instead of database for cost efficiency

**MVP Approach**: Start with OHLCV data only, add raw data storage later if advanced backtesting requires it.

### Technology Stack

The platform will be built using the following technologies:

- **Backend Framework**: Python 3.10+ with Dash (includes built-in Flask server for REST API endpoints)
- **Database**: PostgreSQL 14+ (with TimescaleDB extension for time-series optimization)
- **Real-time Messaging**: Redis (for pub/sub messaging between components)
- **Frontend**: Dash with Plotly (for visualization and control interface) and Mantine UI components
- **Configuration**: JSON files for strategy parameters and bot configurations
- **Deployment**: Docker container setup for development and production

### API Design

**Dash Callbacks**: Real-time updates and user interactions
**REST Endpoints**: Historical data queries for backtesting and analysis
```python
# Built-in Flask routes for historical data
@app.server.route('/api/bot/<bot_id>/trades')
@app.server.route('/api/market/<symbol>/history')  
@app.server.route('/api/backtest/results/<test_id>')
```

### Data Flow

The data flow follows a simple pattern to ensure efficient processing:

1. **Market Data Collection**:
   - Collector fetches data from exchange APIs
   - Raw data is stored in PostgreSQL
   - Processed data (e.g., OHLCV candles) are calculated and stored
   - Real-time updates are published to Redis channels

2. **Signal Generation**:
   - Bots subscribe to relevant data channels and generate signals based on the strategy
   - Signals are stored in database and published to Redis
   
3. **Trade Execution**:
   - Bot manager receives signals from strategies
   - Validates signals against bot parameters and portfolio
   - Simulates or executes trades based on configuration
   - Stores trade information in database

4. **Visualization**:
   - Dashboard subscribes to real-time data and trading updates
   - Queries historical data for charts and performance metrics
   - Provides interface for bot management and configuration

## Development Roadmap

### Phase 1: Foundation (Days 1-5)

**Objective**: Establish core system components and data flow

1. **Day 1-2**: Database Setup and Data Collection
   - Set up PostgreSQL with initial schema
   - Implement OKX API connector
   - Create data storage and processing logic

2. **Day 3-4**: Strategy Engine and Bot Manager
   - Develop strategy interface and 1-2 example strategies
   - Create bot manager with basic controls
   - Implement Redis for real-time messaging

3. **Day 5**: Basic Visualization
   - Set up Dash/Plotly for simple charts
   - Create basic dashboard layout
   - Connect to real-time data sources
   - Create mockup strategies and bots

### Phase 2: Core Functionality (Days 6-10)

**Objective**: Complete essential features for strategy testing

1. **Day 6-7**: Backtesting Engine
   - Get historical data from the database or file (have for BTC/USDT in csv format)
   - Create performance calculation metrics
   - Develop strategy comparison tools

2. **Day 8-9**: Trading Logic
   - Implement virtual trading capability
   - Create trade execution logic
   - Develop portfolio tracking

3. **Day 10**: Dashboard Enhancement
   - Improve visualization components
   - Add bot control interface
   - Implement real-time performance monitoring

### Phase 3: Refinement (Days 11-14)

**Objective**: Polish system and prepare for ongoing use

1. **Day 11-12**: Testing and Debugging
   - Comprehensive system testing
   - Fix identified issues
   - Performance optimization

2. **Day 13-14**: Documentation and Deployment
   - Create user documentation
   - Prepare deployment process
   - Set up basic monitoring

## Technical Considerations

### Scalability Path

While the initial system is designed as a monolithic application for rapid development, several considerations ensure future scalability:

1. **Module Separation**: Clear boundaries between components enable future extraction into microservices
2. **Database Design**: Schema supports partitioning and sharding for larger data volumes
3. **Message Queue**: Redis implementation paves way for more robust messaging (Kafka/RabbitMQ)
4. **API-First Design**: Internal components communicate through well-defined interfaces

### Time Aggregation

Special attention is given to time aggregation to ensure consistency with exchanges:

```python
def aggregate_candles(trades, timeframe, alignment='right'):
    """
    Aggregate trade data into OHLCV candles with consistent timestamp alignment.
    
    Parameters:
    - trades: List of trade dictionaries with timestamp and price
    - timeframe: String representing the timeframe (e.g., '1m', '5m', '1h')
    - alignment: String indicating timestamp alignment ('right' or 'left')
    
    Returns:
    - Dictionary with OHLCV data
    """
    # Convert timeframe to pandas offset
    if timeframe.endswith('m'):
        offset = pd.Timedelta(minutes=int(timeframe[:-1]))
    elif timeframe.endswith('h'):
        offset = pd.Timedelta(hours=int(timeframe[:-1]))
    elif timeframe.endswith('d'):
        offset = pd.Timedelta(days=int(timeframe[:-1]))
    
    # Create DataFrame from trades
    df = pd.DataFrame(trades)
    
    # Convert timestamps to pandas datetime
    df['timestamp'] = pd.to_datetime(df['timestamp'], unit='ms')
    
    # Floor timestamps to timeframe
    if alignment == 'right':
        df['candle_time'] = df['timestamp'].dt.floor(offset)
    else:
        df['candle_time'] = df['timestamp'].dt.ceil(offset) - offset
    
    # Aggregate to OHLCV
    candles = df.groupby('candle_time').agg({
        'price': ['first', 'max', 'min', 'last'],
        'amount': 'sum'
    }).reset_index()
    
    # Rename columns
    candles.columns = ['timestamp', 'open', 'high', 'low', 'close', 'volume']
    
    return candles
```

### Performance Optimization

For the initial release, several performance optimizations are implemented:

1. **Database Indexing**: Proper indexes on timestamp and symbol fields
2. **Query Optimization**: Prepared statements and efficient query patterns
3. **Connection Pooling**: Database connection management to prevent leaks
4. **Data Aggregation**: Pre-calculation of common time intervals
5. **Memory Management**: Proper cleanup of data objects after processing

## User Interface

The initial user interface focuses on functionality over aesthetics, providing essential controls and visualizations, minimalistic design.

1. **Market Data View**
   - Real-time price charts for monitored symbols
   - Order book visualization
   - Recent trades list

2. **Bot Management**
   - Create/configure bot interface
   - Start/stop controls
   - Status indicators

3. **Strategy Dashboard**
   - Strategy selection and configuration
   - Signal visualization
   - Performance metrics

4. **Backtesting Interface**
   - Historical data selection
   - Strategy parameter configuration
   - Results visualization

## Risk Management & Mitigation

### Technical Risks
**Risk:** Exchange API rate limiting affecting data collection
**Mitigation:** Implement intelligent rate limiting, multiple API keys, and fallback data sources

**Risk:** Database performance degradation with large datasets
**Mitigation:** Implement data partitioning, archival strategies, and query optimization (in future)

**Risk:** System downtime during market volatility
**Mitigation:** Design redundant systems, implement circuit breakers, and emergency procedures (in future)

### Business Risks
**Risk:** Regulatory changes affecting crypto trading
**Mitigation:** Implement compliance monitoring, maintain regulatory awareness, design for adaptability

**Risk:** Competition from established trading platforms
**Mitigation:** Focus on unique value propositions, rapid feature development, strong user experience

### 8.3 User Risks
**Risk:** User losses due to platform errors
**Mitigation:** Comprehensive testing, simulation modes, risk warnings, and liability disclaimers

## Future Expansion

While keeping the initial implementation simple, the design accommodates future enhancements:

1. **Authentication System**: Add multi-user support with role-based access
2. **Advanced Strategies**: Support for machine learning and AI-based strategies
3. **Multi-Exchange Support**: Expand beyond OKX to other exchanges
4. **Microservices Migration**: Extract components into separate services
5. **Advanced Monitoring**: Integration with Prometheus/Grafana
6. **Cloud Deployment**: Support for AWS/GCP/Azure deployment

## Success Metrics

The platform's success will be measured by these key metrics:

1. **Development Timeline**: Complete core functionality within 14 days
2. **System Stability**: Maintain 99% uptime during internal testing. System should monitor itself and restart if needed (all or just modules)
3. **Strategy Testing**: Successfully backtest at least 3 different strategies
4. **Bot Performance**: Run at least 2 bots concurrently for 72+ hours