TCPDashboard/docs/1 crypto-bot-prd.md

# Simplified Crypto Trading Bot Platform: Product Requirements Document (PRD)

**Version:** 1.0  
**Date:** May 30, 2025  
**Author:** Vasily
**Status:** Draft  

## Executive Summary

This PRD outlines the development of a simplified crypto trading bot platform that enables strategy testing, development, and execution without the complexity of microservices and advanced monitoring. The goal is to create a functional system within 1-2 weeks that allows for strategy testing while establishing a foundation that can scale in the future. The platform addresses key requirements including data collection, strategy execution, visualization, and backtesting capabilities in a monolithic architecture optimized for internal use.

## Current Requirements & Constraints

- **Speed to Deployment**: System must be functional within 1-2 weeks
- **Scale**: Support for ~10 
- **Architecture**: Monolithic application instead of microservices
- **User Access**: Internal use only initially (no multi-user authentication)
- **Infrastructure**: Simplified deployment without Kubernetes/Docker Swarm
- **Monitoring**: Basic logging for modules

## System Architecture

### High-Level Architecture

The platform will follow a monolithic architecture pattern to enable rapid development while providing clear separation between components:

### Component Details and Functional Requirements

1. **Data Collection Module**
   - Connect to exchange APIs (OKX initially)
   - Retrieve market data (order books, trades, candles)
   - Store raw and processed data in database
   - Send real-time updates through Redis
   - Time Series Data Management (similiar to market standard)

   **FR-001: Unified Data Provider Interface**
   - Support multiple exchanges through standardized adapters
   - Real-time data collection with WebSocket connections
   - Raw data storage for audit and replay capabilities
   - Data validation and error handling mechanisms

   **FR-002: Market Data Processing**
   - OHLCV aggregation with configurable timeframes
   - Technical indicator calculation (SMA, EMA, RSI, MACD, Bollinger Bands)
   - Data normalization across different exchanges
   - Time alignment following exchange standards (right-aligned candles)

2. **Strategy Engine**
   - Provide unified interface for all trading strategies
   - Support multiple strategy types with common parameter structure
   - Generate trading signals based on market data
   - Log strategy performance and signals
   - Strategy implementation as a class.

      **FR-003: Strategy Framework**
      - Base strategy class with standardized interface
      - Support for multiple strategy types
      - Parameter configuration and optimization tools (JSON for the parameters)
      - Signal generation with confidence scoring

      **FR-004: Signal Processing**
      - Real-time signal calculation and validation
      - Signal persistence for analysis and debugging
      - Multi-timeframe analysis capabilities
      - Custom indicator development support

3. **Bot Manager**
   - Create and manage up to 10 concurrent trading bots
   - Configure bot parameters and associated strategies
   - Start/stop individual bots
   - Track bot status and performance

      **FR-005: Bot Lifecycle Management**
      - Bot creation with strategy and parameter selection
      - Start/stop/pause functionality with state persistence
      - Configuration management
      - Resource allocation and monitoring (in future)

      **FR-006: Portfolio Management**
      - Position tracking and balance management
      - Risk management controls (stop-loss, take-profit, position sizing)
      - Multi-bot coordination and conflict resolution (in future)
      - Real-time portfolio valuation (in future)

5. **Trading Execution**
   - Simulate or execute trades based on configuration
   - Stores trade information in database

   **FR-007: Order Management**
   - Order placement with multiple order types (market, limit, stop)
   - Order tracking and status monitoring (in future)
   - Execution confirmation and reconciliation (in future)
   - Fee calculation and tracking (in future)

   **FR-008: Risk Controls**
   - Pre-trade risk validation
   - Position limits and exposure controls (in future)
   - Emergency stop mechanisms (in future)
   - Compliance monitoring and reporting (in future)

4. **Database (PostgreSQL)**
   - Store market data, bot configurations, and trading history
   - Optimized schema for time-series data without complexity
   - Support for data querying and aggregation
   **Database (JSON)**
   - Store strategy parameters and bot onfiguration in JSON in the beginning for simplicity of editing and testing

5. **Backtesting Engine**
   - Run simulations on historical data (batch processing for the signals and incremental processing for the trades)
   - Calculate performance metrics
   - Compare multiple strategies
   - Visualize backtest results

   **FR-009: Historical Simulation**
   - Strategy backtesting on historical data
   - Performance metric calculation (Sharpe ratio, drawdown, win rate)
   - Parameter optimization and sensitivity analysis (in future)
   - Comparison tools for multiple strategies (in future)

   **FR-010: Simulation Engine**
   - Realistic order execution simulation
   - Fee and slippage modeling (in future)
   - Look-ahead bias prevention
   - Parallel backtesting for multiple parameter sets (in future)

6. **Dashboard & Visualization**
   - Display real-time market data
   - Show bot status and performance
   - Visualize strategy signals and executions
   - Provide control interface for bot management

   **FR-011: Dashboard Interface**
   - Real-time bot monitoring and status display
   - Performance charts and metrics visualization
   - Bot configuration and management forms
   - Alert and notification system

   **FR-012: Data Visualization**
   - Interactive price charts with bot signals overlay
   - Performance comparison charts
   - Portfolio allocation and risk metrics display (in future)
   - Historical analysis tools

7. **Web API Service** 
   - REST API for frontend interactions

### Non-Functional Requirements

1 Performance Requirements
**NFR-001: Latency**
- Market data processing: <100ms from exchange to database
- Signal generation: <500ms for standard strategies
- API response time: <200ms for 95% of requests
- Dashboard updates: <2 seconds for real-time data

**NFR-002: Scalability**
- Database queries scalable to 1M+ records per table
- Horizontal scaling capability for all services (in future)

2. Reliability Requirements
**NFR-003: Availability**
- System uptime: 99.5% excluding planned maintenance
- Data collection: 99.9% uptime during market hours
- Automatic failover for critical services
- Graceful degradation during partial outages

**NFR-004: Data Integrity**
- Zero data loss for executed trades
- Transactional consistency for all financial operations
- Regular database backups with point-in-time recovery
- Data validation and error correction mechanisms

3. Security Requirements
**NFR-005: Authentication & Authorization** (in future)

**NFR-006: Data Protection**
- End-to-end encryption for sensitive data (in future)
- Secure storage of API keys and credentials
- Regular security audits and penetration testing (in future)
- Compliance with financial data protection regulations (in future)

## Technical Implementation

### Database Schema

The database schema is designed to be simple yet effective for storing market data, bot configurations, and trading history.

```sql
-- Raw Market Data Tables
CREATE TABLE raw_market_data (
    id SERIAL PRIMARY KEY,
    exchange VARCHAR(50) NOT NULL,
    symbol VARCHAR(20) NOT NULL,
    timestamp TIMESTAMPTZ NOT NULL,
    data JSONB NOT NULL,
    UNIQUE(exchange, symbol, timestamp)
);
CREATE INDEX idx_raw_market_data_symbol_timestamp ON raw_market_data(symbol, timestamp);


-- Processed OHLCV data
CREATE TABLE processed_market_data (
    id SERIAL PRIMARY KEY,
    exchange VARCHAR(50) NOT NULL,
    symbol VARCHAR(20) NOT NULL,
    timeframe VARCHAR(5) NOT NULL,
    timestamp TIMESTAMPTZ NOT NULL,
    open DECIMAL(18,8) NOT NULL,
    high DECIMAL(18,8) NOT NULL,
    low DECIMAL(18,8) NOT NULL,
    close DECIMAL(18,8) NOT NULL,
    volume DECIMAL(18,8) NOT NULL,
    UNIQUE(exchange, symbol, timeframe, timestamp)
) PARTITION BY RANGE (timestamp);
CREATE INDEX idx_processed_market_data_lookup ON processed_market_data(exchange, symbol, timeframe, timestamp);

-- Strategy table
CREATE TABLE strategies (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100) NOT NULL,
    description TEXT,
    parameters JSONB NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

-- Bot table
CREATE TABLE bots (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100) NOT NULL,
    strategy_id INTEGER REFERENCES strategies(id),
    symbol VARCHAR(20) NOT NULL,
    timeframe VARCHAR(5) NOT NULL,
    parameters JSONB NOT NULL,
    status VARCHAR(20) NOT NULL DEFAULT 'inactive',
    last_run TIMESTAMPTZ,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

-- Trading Data Tables
CREATE TABLE signals (
    id SERIAL PRIMARY KEY,
    bot_id INTEGER REFERENCES bots(id),
    timestamp TIMESTAMPTZ NOT NULL,
    symbol VARCHAR(20) NOT NULL,
    signal_type VARCHAR(10) NOT NULL,
    price DECIMAL(18,8),
    confidence DECIMAL(5,4),
    metadata JSONB,
    created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE INDEX idx_signals_bot_time ON signals(bot_id, timestamp);

CREATE TABLE trades (
    id SERIAL PRIMARY KEY,
    bot_id INTEGER REFERENCES bots(id),
    signal_id INTEGER REFERENCES signals(id),
    timestamp TIMESTAMPTZ NOT NULL,
    symbol VARCHAR(20) NOT NULL,
    order_type VARCHAR(20) NOT NULL,
    side VARCHAR(5) NOT NULL,
    price DECIMAL(18,8) NOT NULL,
    quantity DECIMAL(18,8) NOT NULL,
    status VARCHAR(20) NOT NULL,
    fees DECIMAL(18,8),
    metadata JSONB,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE INDEX idx_trades_bot_time ON trades(bot_id, timestamp);

-- Bot Portfolio Tracking
CREATE TABLE bot_performance (
    id SERIAL PRIMARY KEY,
    bot_id INTEGER REFERENCES bots(id),
    timestamp TIMESTAMPTZ NOT NULL,
    equity DECIMAL(18,8) NOT NULL,
    balance DECIMAL(18,8) NOT NULL,
    open_positions JSONB,
    metrics JSONB,
    created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE INDEX idx_bot_performance_bot_time ON bot_performance(bot_id, timestamp);
```

### Technology Stack

The platform will be built using the following technologies:

- **Backend**: Python 3.10+ (FastAPI or Flask)
- **Database**: PostgreSQL 14+ (with TimescaleDB extension for time-series optimization)
- **Real-time Messaging**: Redis (for pub/sub messaging)
- **Frontend**: Dash with Plotly (for visualization and control interface) and Mantine UI (for the dashboard)
- **Deployment**: Simple Docker container setup (not Kubernetes)

### Data Flow

The data flow follows a simple pattern to ensure efficient processing:

1. **Market Data Collection**:
   - Collector fetches data from exchange APIs
   - Raw data is stored in PostgreSQL
   - Processed data (e.g., OHLCV candles) are calculated and stored
   - Real-time updates are published to Redis channels

2. **Signal Generation**:
   - Bots subscribe to relevant data channels and generate signals based on the strategy
   - Signals are stored in database and published to Redis
   
3. **Trade Execution**:
   - Bot manager receives signals from strategies
   - Validates signals against bot parameters and portfolio
   - Simulates or executes trades based on configuration
   - Stores trade information in database

4. **Visualization**:
   - Dashboard subscribes to real-time data and trading updates
   - Queries historical data for charts and performance metrics
   - Provides interface for bot management and configuration

## Development Roadmap

### Phase 1: Foundation (Days 1-5)

**Objective**: Establish core system components and data flow

1. **Day 1-2**: Database Setup and Data Collection
   - Set up PostgreSQL with initial schema
   - Implement OKX API connector
   - Create data storage and processing logic

2. **Day 3-4**: Strategy Engine and Bot Manager
   - Develop strategy interface and 1-2 example strategies
   - Create bot manager with basic controls
   - Implement Redis for real-time messaging

3. **Day 5**: Basic Visualization
   - Set up Dash/Plotly for simple charts
   - Create basic dashboard layout
   - Connect to real-time data sources
   - Create mockup strategies and bots

### Phase 2: Core Functionality (Days 6-10)

**Objective**: Complete essential features for strategy testing

1. **Day 6-7**: Backtesting Engine
   - Get historical data from the database or file (have for BTC/USDT in csv format)
   - Create performance calculation metrics
   - Develop strategy comparison tools

2. **Day 8-9**: Trading Logic
   - Implement virtual trading capability
   - Create trade execution logic
   - Develop portfolio tracking

3. **Day 10**: Dashboard Enhancement
   - Improve visualization components
   - Add bot control interface
   - Implement real-time performance monitoring

### Phase 3: Refinement (Days 11-14)

**Objective**: Polish system and prepare for ongoing use

1. **Day 11-12**: Testing and Debugging
   - Comprehensive system testing
   - Fix identified issues
   - Performance optimization

2. **Day 13-14**: Documentation and Deployment
   - Create user documentation
   - Prepare deployment process
   - Set up basic monitoring

## Technical Considerations

### Scalability Path

While the initial system is designed as a monolithic application for rapid development, several considerations ensure future scalability:

1. **Module Separation**: Clear boundaries between components enable future extraction into microservices
2. **Database Design**: Schema supports partitioning and sharding for larger data volumes
3. **Message Queue**: Redis implementation paves way for more robust messaging (Kafka/RabbitMQ)
4. **API-First Design**: Internal components communicate through well-defined interfaces

### Time Aggregation

Special attention is given to time aggregation to ensure consistency with exchanges:

```python
def aggregate_candles(trades, timeframe, alignment='right'):
    """
    Aggregate trade data into OHLCV candles with consistent timestamp alignment.
    
    Parameters:
    - trades: List of trade dictionaries with timestamp and price
    - timeframe: String representing the timeframe (e.g., '1m', '5m', '1h')
    - alignment: String indicating timestamp alignment ('right' or 'left')
    
    Returns:
    - Dictionary with OHLCV data
    """
    # Convert timeframe to pandas offset
    if timeframe.endswith('m'):
        offset = pd.Timedelta(minutes=int(timeframe[:-1]))
    elif timeframe.endswith('h'):
        offset = pd.Timedelta(hours=int(timeframe[:-1]))
    elif timeframe.endswith('d'):
        offset = pd.Timedelta(days=int(timeframe[:-1]))
    
    # Create DataFrame from trades
    df = pd.DataFrame(trades)
    
    # Convert timestamps to pandas datetime
    df['timestamp'] = pd.to_datetime(df['timestamp'], unit='ms')
    
    # Floor timestamps to timeframe
    if alignment == 'right':
        df['candle_time'] = df['timestamp'].dt.floor(offset)
    else:
        df['candle_time'] = df['timestamp'].dt.ceil(offset) - offset
    
    # Aggregate to OHLCV
    candles = df.groupby('candle_time').agg({
        'price': ['first', 'max', 'min', 'last'],
        'amount': 'sum'
    }).reset_index()
    
    # Rename columns
    candles.columns = ['timestamp', 'open', 'high', 'low', 'close', 'volume']
    
    return candles
```

### Performance Optimization

For the initial release, several performance optimizations are implemented:

1. **Database Indexing**: Proper indexes on timestamp and symbol fields
2. **Query Optimization**: Prepared statements and efficient query patterns
3. **Connection Pooling**: Database connection management to prevent leaks
4. **Data Aggregation**: Pre-calculation of common time intervals
5. **Memory Management**: Proper cleanup of data objects after processing

## User Interface

The initial user interface focuses on functionality over aesthetics, providing essential controls and visualizations, minimalistic design.

1. **Market Data View**
   - Real-time price charts for monitored symbols
   - Order book visualization
   - Recent trades list

2. **Bot Management**
   - Create/configure bot interface
   - Start/stop controls
   - Status indicators

3. **Strategy Dashboard**
   - Strategy selection and configuration
   - Signal visualization
   - Performance metrics

4. **Backtesting Interface**
   - Historical data selection
   - Strategy parameter configuration
   - Results visualization

## Risk Management & Mitigation

### Technical Risks
**Risk:** Exchange API rate limiting affecting data collection
**Mitigation:** Implement intelligent rate limiting, multiple API keys, and fallback data sources

**Risk:** Database performance degradation with large datasets
**Mitigation:** Implement data partitioning, archival strategies, and query optimization (in future)

**Risk:** System downtime during market volatility
**Mitigation:** Design redundant systems, implement circuit breakers, and emergency procedures (in future)

### Business Risks
**Risk:** Regulatory changes affecting crypto trading
**Mitigation:** Implement compliance monitoring, maintain regulatory awareness, design for adaptability

**Risk:** Competition from established trading platforms
**Mitigation:** Focus on unique value propositions, rapid feature development, strong user experience

### 8.3 User Risks
**Risk:** User losses due to platform errors
**Mitigation:** Comprehensive testing, simulation modes, risk warnings, and liability disclaimers

## Future Expansion

While keeping the initial implementation simple, the design accommodates future enhancements:

1. **Authentication System**: Add multi-user support with role-based access
2. **Advanced Strategies**: Support for machine learning and AI-based strategies
3. **Multi-Exchange Support**: Expand beyond OKX to other exchanges
4. **Microservices Migration**: Extract components into separate services
5. **Advanced Monitoring**: Integration with Prometheus/Grafana
6. **Cloud Deployment**: Support for AWS/GCP/Azure deployment

## Success Metrics

The platform's success will be measured by these key metrics:

1. **Development Timeline**: Complete core functionality within 14 days
2. **System Stability**: Maintain 99% uptime during internal testing. System should monitor itself and restart if needed (all or just modules)
3. **Strategy Testing**: Successfully backtest at least 3 different strategies
4. **Bot Performance**: Run at least 2 bots concurrently for 72+ hours