TCPDashboard/CONTEXT.md

Project Context: Advanced Crypto Trading Bot Platform

This document provides a comprehensive overview of the project's architecture, technology stack, conventions, and current implementation status, following the guidelines in context-management.md.

1. Architecture Overview

The platform is a monolithic application built with Python, designed for professional crypto trading strategy development and testing. The architecture features a mature, modular design with production-ready components and clear separation between infrastructure and business logic.

Core Components

• Production-Ready Data Collection Service: A highly sophisticated, multi-process service with robust error handling, health monitoring, and auto-restart capabilities. Features a BaseDataCollector abstraction, CollectorManager for coordination, and exchange-specific implementations (OKX). Includes comprehensive telemetry, state management, and connection management.

• Enterprise Database Layer: PostgreSQL with TimescaleDB support, featuring a mature repository pattern implementation. Uses SQLAlchemy ORM with proper connection pooling, session management, and Alembic migrations. Includes specialized repositories for different entities with consistent error handling.

• Advanced Dashboard & Visualization: A sophisticated Dash application with multiple specialized layouts, real-time charting, technical indicator overlays, and comprehensive system monitoring. Features advanced chart configurations, strategy visualization tools, and responsive UI components.

• Technical Indicators Engine: Complete implementation of technical indicators (SMA, EMA, RSI, MACD, Bollinger Bands) with sparse data handling, chart layer integration, and configurable parameters.

• Real-time Messaging Infrastructure: Redis-based pub/sub system for component communication, with organized channel structures and both synchronous and asynchronous support.

• Strategy Engine: (✅ IMPLEMENTED) Core trading logic components for strategy execution, including base strategy interface, dynamic loading, signal generation, batch processing, and real-time execution.

• Bot Manager: (⚠️ NOT YET IMPLEMENTED) Core trading logic components for bot lifecycle management, and virtual portfolio tracking.

• Backtesting Engine: (⚠️ NOT YET IMPLEMENTED) Historical strategy testing capabilities.

Data Flow

1. Data Collection: DataCollectionService connects to exchange APIs (OKX WebSocket)
2. Processing: Raw trade data processed by exchange-specific processors
3. Aggregation: Trades aggregated into OHLCV candles across multiple timeframes
4. Storage: Both raw trade data and processed OHLCV stored in PostgreSQL
5. Distribution: Real-time data distributed via Redis pub/sub channels
6. Visualization: Dashboard reads from database for charts and system monitoring
7. Strategy Processing: Strategy engine now consumes OHLCV data for signal generation and execution.

System Health & Monitoring

The platform includes comprehensive system health monitoring with:

• Real-time data collection status tracking
• Database performance metrics
• Redis connection monitoring
• CPU and memory usage tracking
• Automatic error detection and alerting

2. Technology Stack

• Backend: Python 3.10+
• Web Framework: Dash with Dash Bootstrap Components and Mantine UI components
• Database: PostgreSQL 14+ with SQLAlchemy ORM and Alembic migrations
• Time-Series: TimescaleDB support (schema available, not yet activated)
• Messaging: Redis for pub/sub communication
• Data Processing: pandas for numerical computations and data manipulation
• Charting: Plotly for advanced financial charts with technical indicators
• Package Management: uv for dependency management
• Containerization: Docker and Docker Compose for development environment
• Testing: pytest with comprehensive test suites for core components
• Logging: Custom unified logging system with component-specific organization

3. Coding Conventions

• Modular Design: Code is organized into modules with a clear purpose (e.g., data, database, dashboard, strategies). See architecture.md for more details.

• Naming Conventions:

  • Classes: PascalCase (e.g., MarketData, BaseDataCollector, CollectorManager)
  • Functions & Methods: snake_case (e.g., get_system_health_layout, connect)
  • Variables & Attributes: snake_case (e.g., exchange_name, _ws_client)
  • Constants: UPPER_SNAKE_CASE (e.g., MAX_RECONNECT_ATTEMPTS)
  • Modules: snake_case.py (e.g., collector_manager.py)
  • Private Attributes/Methods: Single underscore _ prefix

• File Organization & Code Structure:

  • Repository Pattern: Database operations centralized through repository classes
  • Factory Pattern: Used for collector creation, strategy loading and management
  • Abstract Base Classes: Well-defined interfaces for extensibility
  • Dependency Injection: Configuration and dependencies injected rather than hardcoded
  • Error Handling: Custom exception hierarchies with proper error context

• Import/Export Patterns:

  • Absolute imports preferred over relative imports
  • Clean public APIs exposed through __init__.py files
  • Grouped imports: Standard library, third-party, local application

• Configuration Management:

  • Pydantic Settings: Type-safe configuration with environment variable support
  • JSON Configuration: Strategy and bot parameters in JSON files
  • Environment Variables: All credentials and deployment settings via .env

• Database Access:

  • Repository Pattern: All database operations through repository layer
  • SQLAlchemy ORM: Exclusive use of ORM for type safety and maintainability
  • Connection Pooling: Sophisticated connection management with monitoring
  • Session Management: Proper session lifecycle with context managers

• Logging & Monitoring:

  • Unified Logging: Component-specific loggers with centralized configuration
  • Health Monitoring: Comprehensive health checks and telemetry
  • Error Tracking: Detailed error context and automated alerts

4. Current Implementation Status

✅ Completed & Production-Ready Features

🏗️ Infrastructure Foundation (95% Complete)

• Database Schema & Models: Complete PostgreSQL schema with all necessary tables, indexes, and relationships
• Repository Layer: Mature repository pattern with proper ORM usage
• Connection Management: Sophisticated connection pooling with health monitoring
• Migration System: Full Alembic setup for schema versioning

📊 Data Collection System (90% Complete)

• BaseDataCollector: Abstract base with health monitoring, auto-restart, telemetry
• CollectorManager: Multi-collector coordination with lifecycle management
• OKX Integration: Production-ready WebSocket implementation with error handling
• Data Processing: Real-time trade processing and OHLCV aggregation
• State Management: Comprehensive collector state tracking and monitoring

🎯 Dashboard & Visualization (85% Complete)

• Multiple Layouts: Market data, system health, bot management, performance dashboards
• Advanced Charting: Real-time candlestick charts with technical indicator overlays
• System Monitoring: Comprehensive real-time system health dashboard
• Technical Indicators: Complete implementation with chart integration
• User Interface: Professional UI with responsive design and intuitive navigation

📈 Technical Analysis (95% Complete)

• Indicator Library: SMA, EMA, RSI, MACD, Bollinger Bands with proper sparse data handling
• Chart Integration: Sophisticated indicator overlay system with configuration options
• Strategy Configurations: Advanced chart strategy templates and examples

🤖 Strategy Engine (95% Complete)

• BaseStrategy Interface: Implemented and serving as a foundation for strategies
• Strategy Implementations: EMA Crossover, MACD, and RSI strategies implemented
• Strategy Factory: Dynamic strategy loading and registration system in place
• Signal Generation: Core trading signal logic implemented with vectorized operations
• Batch Processing: Capabilities for backtesting large datasets
• Real-time Execution: Event-driven real-time signal generation and broadcasting

🔧 Supporting Systems (90% Complete)

• Logging System: Unified, component-specific logging with automatic cleanup
• Configuration Management: Type-safe settings with environment variable support
• Redis Integration: Pub/sub messaging system for real-time communication
• Development Tools: Comprehensive development and monitoring scripts

⚠️ Critical Gaps - Core Business Logic Missing

🎮 Bot Management System (10% Complete)

• Bot Database Models: Exist but no business logic implementation
• BotManager Class: Does not exist (bot_manager.py missing)
• Virtual Portfolio: No portfolio tracking or P&L calculation
• Bot Lifecycle: No start/stop/monitor functionality

📊 Backtesting Engine (0% Complete)

• BacktestingEngine: No historical testing capabilities
• Performance Metrics: No Sharpe ratio, drawdown, or performance calculation
• Historical Data Processing: No vectorized backtesting implementation

⚡ Real-Time Trading Simulation (0% Complete)

• Trade Execution: No simulated trade processing
• Signal Processing: No real-time signal generation from market data
• Portfolio Updates: No virtual portfolio management

🔧 Technical Debt & Issues

📋 Testing Issues

• Import errors in test suite due to file structure changes (mostly resolved with recent updates)
• Test coverage needs update for new components

📄 Documentation Gaps

• API documentation needs expansion

🎯 Next Phase Priority

The project has excellent infrastructure and a nearly complete Strategy Engine. The next phases will focus on:

1. Chart Integration and Visualization (Task Group 5.0) - Implement live strategy signal visualization layers and performance metrics display.
2. Bot Management System (Task Group 6.0) - Create bot lifecycle management and virtual portfolios
3. Backtesting Engine (Task Group 7.0) - Build historical strategy testing capabilities
4. Real-Time Simulation (Task Group 8.0) - Implement live strategy execution with virtual trading

The sophisticated infrastructure provides a solid foundation for rapid development of the trading logic components.