Ajasra 96ee25bd01 Refactor data validation module for improved modularity and functionality
- Removed the existing `validation.py` file and replaced it with a modular structure, introducing separate files for validation results, field validators, and the base validator class.
- Implemented comprehensive validation functions for common data types, enhancing reusability and maintainability.
- Added a new `__init__.py` to expose the validation utilities, ensuring a clean public interface.
- Created detailed documentation for the validation module, including usage examples and architectural details.
- Introduced extensive unit tests to cover the new validation framework, ensuring reliability and preventing regressions.

These changes enhance the overall architecture of the data validation module, making it more scalable and easier to manage.
2025-06-07 12:31:47 +08:00

10 KiB

Modules Documentation

This section contains detailed technical documentation for all system modules in the TCP Dashboard platform.

📋 Contents

User Interface & Visualization

  • Chart System (charts/) - Comprehensive modular chart system
    • Strategy-driven Configuration: 5 professional trading strategies with JSON persistence
    • 26+ Indicator Presets: SMA, EMA, RSI, MACD, Bollinger Bands with customization
    • User Indicator Management: Interactive CRUD system with real-time updates
    • Modular Dashboard Integration: Separated layouts, callbacks, and components
    • Validation System: 10+ validation rules with detailed error reporting
    • Extensible Architecture: Foundation for bot signal integration
    • Real-time chart updates with indicator toggling
    • Strategy dropdown with auto-loading configurations

Data Collection System

  • [Data Collectors (data_collectors.md)] - Comprehensive guide to the enhanced data collector system

    • BaseDataCollector abstract class with health monitoring
    • CollectorManager for centralized management
    • Exchange Factory Pattern for standardized collector creation
    • Modular Exchange Architecture for scalable implementation
    • Auto-restart and failure recovery mechanisms
    • Health monitoring and alerting systems
    • Performance optimization techniques
    • Integration examples and patterns
    • Comprehensive troubleshooting guide
  • [Data Validation (validation.md)] - Robust data validation framework

    • BaseDataValidator abstract class for exchange-specific validation
    • Field Validators for common market data fields
    • Validation Results with error and warning handling
    • Exchange-Specific Validators with custom rules
    • Comprehensive test coverage
    • Error handling and sanitization
    • Performance optimization for high-frequency validation
    • Integration examples and patterns

Database Operations

  • [Database Operations (database_operations.md)] - Repository pattern for clean database interactions
    • Repository Pattern implementation for data access abstraction
    • MarketDataRepository for candle/OHLCV operations
    • RawTradeRepository for WebSocket data storage
    • Automatic transaction management and session cleanup
    • Configurable duplicate handling with force update options
    • Custom error handling with DatabaseOperationError
    • Database health monitoring and performance statistics
    • Migration guide from direct SQL to repository pattern

Technical Analysis

  • [Technical Indicators (technical-indicators.md)] - Comprehensive technical analysis module
    • Five Core Indicators: SMA, EMA, RSI, MACD, and Bollinger Bands
    • Sparse Data Handling: Optimized for the platform's aggregation strategy
    • Vectorized Calculations: High-performance pandas and numpy implementation
    • Flexible Configuration: JSON-based parameter configuration with validation
    • Integration Ready: Seamless integration with OHLCV data and real-time processing
    • Batch processing for multiple indicators
    • Support for different price columns (open, high, low, close)
    • Comprehensive unit testing and documentation

Logging & Monitoring

  • [Enhanced Logging System (logging.md)] - Unified logging framework
    • Multi-level logging with automatic cleanup
    • Console and file output with formatting
    • Performance monitoring integration
    • Cross-component logging standards
    • Log aggregation and analysis

🔧 Component Architecture

Core Components

┌─────────────────────────────────────────────────────────────┐
│                   TCP Dashboard Platform                    │
│                                                             │
│  ┌─────────────────────────────────────────────────────┐    │
│  │              Modular Dashboard System               │    │
│  │  • Separated layouts, callbacks, components        │    │
│  │  • Chart layers with strategy management           │    │
│  │  • Real-time indicator updates                     │    │
│  │  • User indicator CRUD operations                  │    │
│  └─────────────────────────────────────────────────────┘    │
│                           │                                 │
│  ┌─────────────────────────────────────────────────────┐    │
│  │              CollectorManager                       │    │
│  │  ┌─────────────────────────────────────────────────┐│    │
│  │  │            Global Health Monitor               ││    │
│  │  │  • System-wide health checks                   ││    │
│  │  │  • Auto-restart coordination                   ││    │
│  │  │  • Performance analytics                       ││    │
│  │  └─────────────────────────────────────────────────┘│    │
│  │                         │                           │    │
│  │ ┌─────────────┐ ┌─────────────┐ ┌────────────────┐  │    │
│  │ │OKX Collector│ │Binance Coll.│ │Custom Collector│  │    │
│  │ │• Health Mon │ │• Health Mon │ │• Health Monitor│  │    │
│  │ │• Auto-restart│ │• Auto-restart│ │• Auto-restart │  │    │
│  │ │• Data Valid │ │• Data Valid │ │• Data Validate │  │    │
│  │ └─────────────┘ └─────────────┘ └────────────────┘  │    │
│  └─────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────┘

Design Patterns

  • Factory Pattern: Standardized component creation across exchanges and charts
  • Observer Pattern: Event-driven data processing and real-time updates
  • Strategy Pattern: Pluggable data processing and chart configuration strategies
  • Singleton Pattern: Centralized logging and configuration management
  • Modular Architecture: Separated concerns with reusable components
  • Repository Pattern: Clean database access abstraction

🚀 Quick Start

Using Chart Components

# Chart system usage
from components.charts.config import get_available_strategy_names
from components.charts.indicator_manager import get_indicator_manager

# Get available strategies
strategy_names = get_available_strategy_names()

# Create custom indicator
manager = get_indicator_manager()
indicator = manager.create_indicator(
    name="Custom SMA 50",
    indicator_type="sma",
    parameters={"period": 50}
)

Using Data Components

# Data Collector usage
from data.exchanges import create_okx_collector
from data.base_collector import DataType

collector = create_okx_collector(
    symbol='BTC-USDT',
    data_types=[DataType.TRADE, DataType.ORDERBOOK]
)

# Logging usage
from utils.logger import get_logger

logger = get_logger("my_component")
logger.info("Component initialized")

Component Integration

# Integrating multiple components
from data.collector_manager import CollectorManager
from dashboard.app import create_app
from utils.logger import get_logger

# Start data collection
manager = CollectorManager("production_system")

# Create dashboard app
app = create_app()

# Components work together seamlessly
await manager.start()
app.run_server(host='0.0.0.0', port=8050)

📊 Performance & Monitoring

Health Monitoring

All components include built-in health monitoring:

  • Real-time Status: Component state and performance metrics
  • Auto-Recovery: Automatic restart on failures
  • Performance Tracking: Message rates, uptime, error rates
  • Alerting: Configurable alerts for component health
  • Dashboard Integration: Visual system health monitoring

Logging Integration

Unified logging across all components:

  • Structured Logging: JSON-formatted logs for analysis
  • Multiple Levels: Debug, Info, Warning, Error levels
  • Automatic Cleanup: Log rotation and old file cleanup
  • Performance Metrics: Built-in performance tracking
  • Component Isolation: Separate loggers for different modules

📈 Future Components

Planned component additions:

  • Signal Layer: Bot trading signal visualization and integration
  • Strategy Engine: Trading strategy execution framework
  • Portfolio Manager: Position and risk management
  • Alert Manager: Advanced alerting and notification system
  • Data Analytics: Historical data analysis and reporting

For the complete documentation index, see the [main documentation README (../README.md)]