TCPDashboard/CONTEXT.md

# Project Context: Advanced Crypto Trading Bot Platform

This document provides a comprehensive overview of the project's architecture, technology stack, conventions, and current implementation status, following the guidelines in `context-management.md`.

## 1. Architecture Overview

The platform is a **monolithic application** built with Python, designed for professional crypto trading strategy development and testing. The architecture features a mature, modular design with production-ready components and clear separation between infrastructure and business logic.

### Core Components

- **Production-Ready Data Collection Service**: A highly sophisticated, multi-process service with robust error handling, health monitoring, and auto-restart capabilities. Features a `BaseDataCollector` abstraction, `CollectorManager` for coordination, and exchange-specific implementations (OKX). Includes comprehensive telemetry, state management, and connection management.

- **Enterprise Database Layer**: PostgreSQL with TimescaleDB support, featuring a mature repository pattern implementation. Uses SQLAlchemy ORM with proper connection pooling, session management, and Alembic migrations. Includes specialized repositories for different entities with consistent error handling.

- **Advanced Dashboard & Visualization**: A sophisticated Dash application with multiple specialized layouts, real-time charting, technical indicator overlays, and comprehensive system monitoring. Features advanced chart configurations, strategy visualization tools, and responsive UI components.

- **Technical Indicators Engine**: Complete implementation of technical indicators (SMA, EMA, RSI, MACD, Bollinger Bands) with sparse data handling, chart layer integration, and configurable parameters.

- **Real-time Messaging Infrastructure**: Redis-based pub/sub system for component communication, with organized channel structures and both synchronous and asynchronous support.

- **Strategy Engine**: (✅ **IMPLEMENTED**) Core trading logic components for strategy execution, including base strategy interface, dynamic loading, signal generation, batch processing, and real-time execution.

- **Bot Manager**: (⚠️ **NOT YET IMPLEMENTED**) Core trading logic components for bot lifecycle management, and virtual portfolio tracking.

- **Backtesting Engine**: (⚠️ **NOT YET IMPLEMENTED**) Historical strategy testing capabilities.

### Data Flow
1. **Data Collection**: `DataCollectionService` connects to exchange APIs (OKX WebSocket)
2. **Processing**: Raw trade data processed by exchange-specific processors  
3. **Aggregation**: Trades aggregated into OHLCV candles across multiple timeframes
4. **Storage**: Both raw trade data and processed OHLCV stored in PostgreSQL
5. **Distribution**: Real-time data distributed via Redis pub/sub channels
6. **Visualization**: Dashboard reads from database for charts and system monitoring
7. **Strategy Processing**: Strategy engine now consumes OHLCV data for signal generation and execution.

### System Health & Monitoring
The platform includes comprehensive system health monitoring with:
- Real-time data collection status tracking
- Database performance metrics
- Redis connection monitoring  
- CPU and memory usage tracking
- Automatic error detection and alerting

## 2. Technology Stack

- **Backend**: Python 3.10+
- **Web Framework**: Dash with Dash Bootstrap Components and Mantine UI components
- **Database**: PostgreSQL 14+ with SQLAlchemy ORM and Alembic migrations
- **Time-Series**: TimescaleDB support (schema available, not yet activated)
- **Messaging**: Redis for pub/sub communication
- **Data Processing**: pandas for numerical computations and data manipulation
- **Charting**: Plotly for advanced financial charts with technical indicators
- **Package Management**: `uv` for dependency management
- **Containerization**: Docker and Docker Compose for development environment
- **Testing**: pytest with comprehensive test suites for core components
- **Logging**: Custom unified logging system with component-specific organization

## 3. Coding Conventions

- **Modular Design**: Code is organized into modules with a clear purpose (e.g., `data`, `database`, `dashboard`, `strategies`). See `architecture.md` for more details.
- **Naming Conventions**:
    - **Classes**: `PascalCase` (e.g., `MarketData`, `BaseDataCollector`, `CollectorManager`)
    - **Functions & Methods**: `snake_case` (e.g., `get_system_health_layout`, `connect`)
    - **Variables & Attributes**: `snake_case` (e.g., `exchange_name`, `_ws_client`)
    - **Constants**: `UPPER_SNAKE_CASE` (e.g., `MAX_RECONNECT_ATTEMPTS`)
    - **Modules**: `snake_case.py` (e.g., `collector_manager.py`)
    - **Private Attributes/Methods**: Single underscore `_` prefix

- **File Organization & Code Structure**:
    - **Repository Pattern**: Database operations centralized through repository classes
    - **Factory Pattern**: Used for collector creation, strategy loading and management
    - **Abstract Base Classes**: Well-defined interfaces for extensibility  
    - **Dependency Injection**: Configuration and dependencies injected rather than hardcoded
    - **Error Handling**: Custom exception hierarchies with proper error context

- **Import/Export Patterns**:
    - **Absolute imports** preferred over relative imports
    - **Clean public APIs** exposed through `__init__.py` files
    - **Grouped imports**: Standard library, third-party, local application

- **Configuration Management**:
    - **Pydantic Settings**: Type-safe configuration with environment variable support
    - **JSON Configuration**: Strategy and bot parameters in JSON files
    - **Environment Variables**: All credentials and deployment settings via `.env`

- **Database Access**: 
    - **Repository Pattern**: All database operations through repository layer
    - **SQLAlchemy ORM**: Exclusive use of ORM for type safety and maintainability
    - **Connection Pooling**: Sophisticated connection management with monitoring
    - **Session Management**: Proper session lifecycle with context managers

- **Logging & Monitoring**:
    - **Unified Logging**: Component-specific loggers with centralized configuration
    - **Health Monitoring**: Comprehensive health checks and telemetry
    - **Error Tracking**: Detailed error context and automated alerts

## 4. Current Implementation Status

### ✅ **Completed & Production-Ready Features**

**🏗️ Infrastructure Foundation (95% Complete)**
- **Database Schema & Models**: Complete PostgreSQL schema with all necessary tables, indexes, and relationships
- **Repository Layer**: Mature repository pattern with proper ORM usage
- **Connection Management**: Sophisticated connection pooling with health monitoring
- **Migration System**: Full Alembic setup for schema versioning

**📊 Data Collection System (90% Complete)**  
- **BaseDataCollector**: Abstract base with health monitoring, auto-restart, telemetry
- **CollectorManager**: Multi-collector coordination with lifecycle management
- **OKX Integration**: Production-ready WebSocket implementation with error handling
- **Data Processing**: Real-time trade processing and OHLCV aggregation
- **State Management**: Comprehensive collector state tracking and monitoring

**🎯 Dashboard & Visualization (85% Complete)**
- **Multiple Layouts**: Market data, system health, bot management, performance dashboards
- **Advanced Charting**: Real-time candlestick charts with technical indicator overlays
- **System Monitoring**: Comprehensive real-time system health dashboard
- **Technical Indicators**: Complete implementation with chart integration
- **User Interface**: Professional UI with responsive design and intuitive navigation

**📈 Technical Analysis (95% Complete)**
- **Indicator Library**: SMA, EMA, RSI, MACD, Bollinger Bands with proper sparse data handling
- **Chart Integration**: Sophisticated indicator overlay system with configuration options
- **Strategy Configurations**: Advanced chart strategy templates and examples

**🤖 Strategy Engine (95% Complete)**
- **BaseStrategy Interface**: Implemented and serving as a foundation for strategies
- **Strategy Implementations**: EMA Crossover, MACD, and RSI strategies implemented
- **Strategy Factory**: Dynamic strategy loading and registration system in place
- **Signal Generation**: Core trading signal logic implemented with vectorized operations
- **Batch Processing**: Capabilities for backtesting large datasets
- **Real-time Execution**: Event-driven real-time signal generation and broadcasting

**🔧 Supporting Systems (90% Complete)**
- **Logging System**: Unified, component-specific logging with automatic cleanup
- **Configuration Management**: Type-safe settings with environment variable support
- **Redis Integration**: Pub/sub messaging system for real-time communication
- **Development Tools**: Comprehensive development and monitoring scripts

### ⚠️ **Critical Gaps - Core Business Logic Missing**

**🎮 Bot Management System (10% Complete)**  
- **Bot Database Models**: Exist but no business logic implementation
- **BotManager Class**: Does not exist (`bot_manager.py` missing)
- **Virtual Portfolio**: No portfolio tracking or P&L calculation
- **Bot Lifecycle**: No start/stop/monitor functionality

**📊 Backtesting Engine (0% Complete)**
- **BacktestingEngine**: No historical testing capabilities
- **Performance Metrics**: No Sharpe ratio, drawdown, or performance calculation
- **Historical Data Processing**: No vectorized backtesting implementation

**⚡ Real-Time Trading Simulation (0% Complete)**
- **Trade Execution**: No simulated trade processing
- **Signal Processing**: No real-time signal generation from market data
- **Portfolio Updates**: No virtual portfolio management

### 🔧 **Technical Debt & Issues**

**📋 Testing Issues**
- Import errors in test suite due to file structure changes (mostly resolved with recent updates)
- Test coverage needs update for new components

**📄 Documentation Gaps**  
- API documentation needs expansion

### 🎯 **Next Phase Priority**

The project has **excellent infrastructure** and a nearly complete **Strategy Engine**. The next phases will focus on:

1. **Chart Integration and Visualization** (Task Group 5.0) - Implement live strategy signal visualization layers and performance metrics display.
2. **Bot Management System** (Task Group 6.0) - Create bot lifecycle management and virtual portfolios  
3. **Backtesting Engine** (Task Group 7.0) - Build historical strategy testing capabilities
4. **Real-Time Simulation** (Task Group 8.0) - Implement live strategy execution with virtual trading

The sophisticated infrastructure provides a solid foundation for rapid development of the trading logic components.