TCPDashboard/docs/logging_system.md

Conditional Logging System

Overview

The TCP Dashboard project implements a sophisticated conditional logging system that provides fine-grained control over logging behavior across all components. This system supports hierarchical logging, conditional logging, and error-only logging modes.

Key Features

1. Conditional Logging

• No Logger: If no logger instance is passed to a component's constructor, that component performs no logging operations
• Logger Provided: If a logger instance is passed, the component uses it for logging
• Error-Only Mode: If log_errors_only=True is set, only error and critical level messages are logged

2. Logger Inheritance

• Components that receive a logger pass the same logger instance down to child components
• This creates a hierarchical logging structure that follows the component hierarchy

3. Hierarchical File Organization

• Log files are organized based on component hierarchy
• Each major component gets its own log directory
• Child components log to their parent's log file

Component Hierarchy

Top-level Application (individual logger)
├── ProductionManager (individual logger)
│   ├── DataSaver (receives logger from ProductionManager)
│   ├── DataValidator (receives logger from ProductionManager)
│   ├── DatabaseConnection (receives logger from ProductionManager)
│   └── CollectorManager (individual logger)
│       ├── OKX collector BTC-USD (individual logger)
│       │   ├── DataAggregator (receives logger from OKX collector)
│       │   ├── DataTransformer (receives logger from OKX collector)
│       │   └── DataProcessor (receives logger from OKX collector)
│       └── Another collector...

Usage Examples

Basic Usage

from utils.logger import get_logger
from data.exchanges.okx.collector import OKXCollector

# Create a logger for the collector
collector_logger = get_logger('okx_collector_btc_usdt', verbose=True)

# Create collector with logger - all child components will use this logger
collector = OKXCollector(
    symbol='BTC-USDT',
    logger=collector_logger
)

# Child components (data processor, validator, transformer) will automatically
# receive and use the same logger instance

No Logging Mode

# Create collector without logger - no logging will be performed
collector = OKXCollector(
    symbol='BTC-USDT',
    logger=None  # or simply omit the parameter
)

# No log files will be created, no console output

Error-Only Logging Mode

from utils.logger import get_logger
from data.collector_manager import CollectorManager

# Create logger for manager
manager_logger = get_logger('collector_manager', verbose=True)

# Create manager with error-only logging
manager = CollectorManager(
    manager_name="production_manager",
    logger=manager_logger,
    log_errors_only=True  # Only errors and critical messages will be logged
)

# Manager will only log errors, but child collectors can have their own loggers

Hierarchical Logging Setup

from utils.logger import get_logger
from data.collector_manager import CollectorManager
from data.exchanges.okx.collector import OKXCollector

# Create manager with its own logger
manager_logger = get_logger('collector_manager', verbose=True)
manager = CollectorManager(logger=manager_logger)

# Create individual collectors with their own loggers
btc_logger = get_logger('okx_collector_btc_usdt', verbose=True)
eth_logger = get_logger('okx_collector_eth_usdt', verbose=True)

btc_collector = OKXCollector('BTC-USDT', logger=btc_logger)
eth_collector = OKXCollector('ETH-USDT', logger=eth_logger)

# Add collectors to manager
manager.add_collector(btc_collector)
manager.add_collector(eth_collector)

# Result:
# - Manager logs to: logs/collector_manager/YYYY-MM-DD.txt
# - BTC collector logs to: logs/okx_collector_btc_usdt/YYYY-MM-DD.txt
# - ETH collector logs to: logs/okx_collector_eth_usdt/YYYY-MM-DD.txt
# - All child components of each collector log to their parent's file

Implementation Details

Base Classes

All base classes support conditional logging:

class BaseDataCollector:
    def __init__(self, ..., logger=None, log_errors_only=False):
        self.logger = logger
        self.log_errors_only = log_errors_only
        
    def _log_debug(self, message: str) -> None:
        if self.logger and not self.log_errors_only:
            self.logger.debug(message)
    
    def _log_error(self, message: str, exc_info: bool = False) -> None:
        if self.logger:
            self.logger.error(message, exc_info=exc_info)

Child Component Pattern

Child components receive logger from parent:

class OKXCollector(BaseDataCollector):
    def __init__(self, symbol: str, logger=None):
        super().__init__(..., logger=logger)
        
        # Pass logger to child components
        self._data_processor = OKXDataProcessor(
            symbol, 
            logger=self.logger  # Pass parent's logger
        )

Conditional Logging Helpers

All components use helper methods for conditional logging:

def _log_debug(self, message: str) -> None:
    """Log debug message if logger is available and not in errors-only mode."""
    if self.logger and not self.log_errors_only:
        self.logger.debug(message)

def _log_info(self, message: str) -> None:
    """Log info message if logger is available and not in errors-only mode."""
    if self.logger and not self.log_errors_only:
        self.logger.info(message)

def _log_warning(self, message: str) -> None:
    """Log warning message if logger is available and not in errors-only mode."""
    if self.logger and not self.log_errors_only:
        self.logger.warning(message)

def _log_error(self, message: str, exc_info: bool = False) -> None:
    """Log error message if logger is available (always logs errors)."""
    if self.logger:
        self.logger.error(message, exc_info=exc_info)

def _log_critical(self, message: str, exc_info: bool = False) -> None:
    """Log critical message if logger is available (always logs critical)."""
    if self.logger:
        self.logger.critical(message, exc_info=exc_info)

Log File Structure

logs/
├── collector_manager/
│   └── 2024-01-15.txt
├── okx_collector_btc_usdt/
│   └── 2024-01-15.txt
├── okx_collector_eth_usdt/
│   └── 2024-01-15.txt
└── production_manager/
    └── 2024-01-15.txt

Configuration Options

Logger Parameters

• logger: Logger instance or None
• log_errors_only: Boolean flag for error-only mode
• verbose: Console output (when creating new loggers)
• clean_old_logs: Automatic cleanup of old log files
• max_log_files: Maximum number of log files to keep

Environment Variables

# Enable verbose console logging
VERBOSE_LOGGING=true

# Enable console output
LOG_TO_CONSOLE=true

Best Practices

1. Component Design

• Always accept logger=None parameter in constructors
• Pass logger to all child components
• Use conditional logging helper methods
• Never assume logger is available

2. Error Handling

• Always log errors regardless of log_errors_only setting
• Use appropriate log levels
• Include context in error messages

3. Performance

• Conditional logging has minimal performance impact
• Logger checks are fast boolean operations
• No string formatting when logging is disabled

4. Testing

• Test components with and without loggers
• Verify error-only mode works correctly
• Check that child components receive loggers properly

Migration Guide

Updating Existing Components

1. Add logger parameter to constructor:

def __init__(self, ..., logger=None, log_errors_only=False):

2. Add conditional logging helpers:

def _log_debug(self, message: str) -> None:
    if self.logger and not self.log_errors_only:
        self.logger.debug(message)

3. Update all logging calls:

# Before
self.logger.info("Message")

# After
self._log_info("Message")

4. Pass logger to child components:

child = ChildComponent(logger=self.logger)

Testing Changes

# Test without logger
component = MyComponent(logger=None)
# Should work without errors, no logging

# Test with logger
logger = get_logger('test_component')
component = MyComponent(logger=logger)
# Should log normally

# Test error-only mode
component = MyComponent(logger=logger, log_errors_only=True)
# Should only log errors

This conditional logging system provides maximum flexibility while maintaining clean, maintainable code that works in all scenarios.