TCPDashboard/docs/README.md

# TCP Dashboard Documentation

Welcome to the **TCP Dashboard** (Trading Crypto Platform) documentation. This platform provides a comprehensive solution for cryptocurrency trading bot development, backtesting, and portfolio management.

## 📚 Documentation Structure

The documentation is organized into specialized sections for better navigation and maintenance:

### 🏗️ **[Architecture & Design](architecture/)**

- **[Architecture Overview](architecture/architecture.md)** - High-level system architecture and component design
- **[Data Processing Refactor](architecture/data-processing-refactor.md)** - *New modular data processing architecture*
  - Common utilities shared across all exchanges
  - Right-aligned timestamp aggregation strategy
  - Future leakage prevention mechanisms
  - Exchange-specific component design
- **[Crypto Bot PRD](architecture/crypto-bot-prd.md)** - Product Requirements Document for the crypto trading bot platform

### 🔧 **[Core Components](components/)**

- **[Data Collectors](components/data_collectors.md)** - *Comprehensive guide to the enhanced data collector system*
  - BaseDataCollector abstract class with health monitoring
  - CollectorManager for centralized management
  - Exchange Factory Pattern for standardized collector creation
  - Modular Exchange Architecture for scalable implementation
  - Auto-restart and failure recovery mechanisms

- **[Technical Indicators](components/technical-indicators.md)** - *Technical analysis module for trading strategies*
  - SMA, EMA, RSI, MACD, and Bollinger Bands calculations
  - Optimized for sparse OHLCV data handling
  - Vectorized calculations using pandas and numpy
  - JSON configuration support with validation
  - Integration with aggregation strategy

- **[Logging System](components/logging.md)** - *Unified logging framework*
  - Multi-level logging with automatic cleanup
  - Console and file output with formatting
  - Performance monitoring integration

### 🌐 **[Exchange Integrations](exchanges/)**

- **[OKX Collector](exchanges/okx_collector.md)** - *Complete guide to OKX exchange integration*
  - Real-time trades, orderbook, and ticker data collection
  - WebSocket connection management with OKX-specific ping/pong
  - Factory pattern usage and configuration
  - Production deployment guide

- **[Exchange Overview](exchanges/)** - Multi-exchange architecture and comparison

### 📖 **[Setup & Guides](guides/)**

- **[Setup Guide](guides/setup.md)** - *Comprehensive setup instructions*
  - Environment configuration and prerequisites
  - Database setup with Docker and PostgreSQL
  - Development workflow and best practices
  - Production deployment guidelines

### 📋 **[Technical Reference](reference/)**

- **[Project Specification](reference/specification.md)** - *Technical specifications and requirements*
  - System requirements and constraints
  - Database schema specifications
  - API endpoint definitions
  - Data format specifications

- **[Aggregation Strategy](reference/aggregation-strategy.md)** - *Comprehensive data aggregation documentation*
  - Right-aligned timestamp strategy (industry standard)
  - Future leakage prevention safeguards
  - Real-time vs historical processing
  - Database storage patterns
  - Testing methodology and examples

## 🎯 **Quick Start**

1. **New to the platform?** Start with the [Setup Guide](guides/setup.md)
2. **Implementing data collectors?** See [Data Collectors Documentation](components/data_collectors.md)
3. **Understanding the architecture?** Read [Architecture Overview](architecture/architecture.md)
4. **Exchange integration?** Check [Exchange Documentation](exchanges/)
5. **Troubleshooting?** Check component-specific documentation

## 🏛️ **System Components**

### Core Infrastructure
- **Database Layer**: PostgreSQL with SQLAlchemy models
- **Real-time Messaging**: Redis pub/sub for data distribution
- **Configuration Management**: Pydantic-based settings
- **Containerization**: Docker and docker-compose setup

### Data Collection & Processing
- **Abstract Base Collectors**: Standardized interface for all exchange connectors
- **Exchange Factory Pattern**: Unified collector creation across exchanges
- **Modular Exchange Architecture**: Organized exchange implementations in dedicated folders
- **Health Monitoring**: Automatic failure detection and recovery
- **Data Validation**: Comprehensive validation for market data
- **Multi-Exchange Support**: OKX (production-ready), Binance and other exchanges (planned)

### Trading & Strategy Engine
- **Strategy Framework**: Base strategy classes and implementations
- **Bot Management**: Lifecycle management with JSON configuration
- **Backtesting Engine**: Historical strategy testing with performance metrics
- **Portfolio Management**: Virtual trading with P&L tracking

### User Interface
- **Dashboard**: Dash-based web interface with Mantine UI
- **Real-time Charts**: Interactive price charts with technical indicators
- **Bot Controls**: Start/stop/configure trading bots
- **Performance Analytics**: Portfolio visualization and trade analytics

## 📋 **Task Progress**

The platform follows a structured development approach with clearly defined tasks:

- ✅ **Database Foundation** - Complete
- ✅ **Enhanced Data Collectors** - Complete with health monitoring
- ✅ **OKX Data Collector** - Complete with factory pattern and production testing
- ⏳ **Multi-Exchange Support** - In progress (Binance connector next)
- ⏳ **Basic Dashboard** - Planned
- ⏳ **Strategy Engine** - Planned
- ⏳ **Advanced Features** - Planned

For detailed task tracking, see [tasks/tasks-crypto-bot-prd.md](../tasks/tasks-crypto-bot-prd.md).

## 🛠️ **Development Workflow**

### Setting Up Development Environment

```bash
# Clone and setup
git clone <repository>
cd TCPDashboard

# Install dependencies with UV
uv sync

# Setup environment
cp .env.example .env
# Edit .env with your configuration

# Start services
docker-compose up -d

# Initialize database
uv run python scripts/init_database.py

# Run tests
uv run pytest
```

### Key Development Tools

- **UV**: Modern Python package management
- **pytest**: Testing framework with async support
- **SQLAlchemy**: Database ORM with migration support
- **Dash + Mantine**: Modern web UI framework
- **Docker**: Containerized development environment

## 🔍 **Testing**

The platform includes comprehensive test coverage:

- **Unit Tests**: Individual component testing
- **Integration Tests**: Cross-component functionality
- **Performance Tests**: Load and stress testing
- **End-to-End Tests**: Full system workflows

```bash
# Run all tests
uv run pytest

# Run specific test files
uv run pytest tests/test_base_collector.py
uv run pytest tests/test_collector_manager.py

# Run with coverage
uv run pytest --cov=data --cov-report=html
```

## 📊 **Monitoring & Observability**

### Logging
- **Structured Logging**: JSON-formatted logs with automatic cleanup
- **Multiple Levels**: Debug, Info, Warning, Error with configurable output
- **Component Isolation**: Separate loggers for different system components

### Health Monitoring
- **Collector Health**: Real-time status and performance metrics
- **Auto-Recovery**: Automatic restart on failures
- **Performance Tracking**: Message rates, uptime, error rates

### Metrics Integration
- **Prometheus Support**: Built-in metrics collection
- **Custom Dashboards**: System performance visualization
- **Alerting**: Configurable alerts for system health

## 🔐 **Security & Best Practices**

### Configuration Management
- **Environment Variables**: All sensitive data via `.env` files
- **No Hardcoded Secrets**: Clean separation of configuration and code
- **Validation**: Pydantic-based configuration validation

### Data Handling
- **Input Validation**: Comprehensive validation for all external data
- **Error Handling**: Robust error handling with proper logging
- **Resource Management**: Proper cleanup and resource management

### Code Quality
- **Type Hints**: Full type annotation coverage
- **Documentation**: Comprehensive docstrings and comments
- **Testing**: High test coverage with multiple test types
- **Code Standards**: Consistent formatting and patterns

## 🤝 **Contributing**

### Development Guidelines
1. Follow existing code patterns and architecture
2. Add comprehensive tests for new functionality
3. Update documentation for API changes
4. Use type hints and proper error handling
5. Follow the existing logging patterns

### Code Review Process
1. Create feature branches from main
2. Write tests before implementing features
3. Ensure all tests pass and maintain coverage
4. Update relevant documentation
5. Submit pull requests with clear descriptions

## 📞 **Support**

### Getting Help
1. **Documentation**: Check relevant component documentation
2. **Logs**: Review system logs in `./logs/` directory  
3. **Status**: Use built-in status and health check methods
4. **Tests**: Run test suite to verify system integrity

### Common Issues
- **Database Connection**: Check Docker services and environment variables
- **Collector Failures**: Review collector health status and logs
- **Performance Issues**: Monitor system resources and optimize accordingly

---

## 📁 **Documentation File Structure**

```
docs/
├── README.md                    # This file - main documentation index
├── architecture/                # System architecture and design
│   ├── README.md               # Architecture overview
│   ├── architecture.md         # Technical architecture
│   └── crypto-bot-prd.md       # Product requirements
├── components/                  # Core system components
│   ├── README.md               # Component overview
│   ├── data_collectors.md      # Data collection system
│   └── logging.md              # Logging framework
├── exchanges/                   # Exchange integrations
│   ├── README.md               # Exchange overview
│   └── okx_collector.md        # OKX implementation
├── guides/                      # User guides and tutorials
│   ├── README.md               # Guide overview
│   └── setup.md                # Setup instructions
└── reference/                   # Technical reference
    ├── README.md               # Reference overview
    └── specification.md        # Technical specifications
```

## 🔗 **Navigation**

- **🏗️ [Architecture & Design](architecture/)** - System design and requirements
- **🔧 [Core Components](components/)** - Technical implementation details
- **🌐 [Exchange Integrations](exchanges/)** - Exchange-specific documentation
- **📖 [Setup & Guides](guides/)** - User guides and tutorials
- **📋 [Technical Reference](reference/)** - API specifications and schemas

---

*Last updated: $(date)*

For the most current information, refer to the individual component documentation linked above.