TCPDashboard/docs/logging_system.md

# Conditional Logging System

## Overview

The TCP Dashboard project implements a sophisticated conditional logging system that provides fine-grained control over logging behavior across all components. This system supports hierarchical logging, conditional logging, and error-only logging modes.

## Key Features

### 1. Conditional Logging
- **No Logger**: If no logger instance is passed to a component's constructor, that component performs no logging operations
- **Logger Provided**: If a logger instance is passed, the component uses it for logging
- **Error-Only Mode**: If `log_errors_only=True` is set, only error and critical level messages are logged

### 2. Logger Inheritance
- Components that receive a logger pass the same logger instance down to child components
- This creates a hierarchical logging structure that follows the component hierarchy

### 3. Hierarchical File Organization
- Log files are organized based on component hierarchy
- Each major component gets its own log directory
- Child components log to their parent's log file

## Component Hierarchy

```
Top-level Application (individual logger)
├── ProductionManager (individual logger)
│   ├── DataSaver (receives logger from ProductionManager)
│   ├── DataValidator (receives logger from ProductionManager)
│   ├── DatabaseConnection (receives logger from ProductionManager)
│   └── CollectorManager (individual logger)
│       ├── OKX collector BTC-USD (individual logger)
│       │   ├── DataAggregator (receives logger from OKX collector)
│       │   ├── DataTransformer (receives logger from OKX collector)
│       │   └── DataProcessor (receives logger from OKX collector)
│       └── Another collector...
```

## Usage Examples

### Basic Usage

```python
from utils.logger import get_logger
from data.exchanges.okx.collector import OKXCollector

# Create a logger for the collector
collector_logger = get_logger('okx_collector_btc_usdt', verbose=True)

# Create collector with logger - all child components will use this logger
collector = OKXCollector(
    symbol='BTC-USDT',
    logger=collector_logger
)

# Child components (data processor, validator, transformer) will automatically
# receive and use the same logger instance
```

### No Logging Mode

```python
# Create collector without logger - no logging will be performed
collector = OKXCollector(
    symbol='BTC-USDT',
    logger=None  # or simply omit the parameter
)

# No log files will be created, no console output
```

### Error-Only Logging Mode

```python
from utils.logger import get_logger
from data.collector_manager import CollectorManager

# Create logger for manager
manager_logger = get_logger('collector_manager', verbose=True)

# Create manager with error-only logging
manager = CollectorManager(
    manager_name="production_manager",
    logger=manager_logger,
    log_errors_only=True  # Only errors and critical messages will be logged
)

# Manager will only log errors, but child collectors can have their own loggers
```

### Hierarchical Logging Setup

```python
from utils.logger import get_logger
from data.collector_manager import CollectorManager
from data.exchanges.okx.collector import OKXCollector

# Create manager with its own logger
manager_logger = get_logger('collector_manager', verbose=True)
manager = CollectorManager(logger=manager_logger)

# Create individual collectors with their own loggers
btc_logger = get_logger('okx_collector_btc_usdt', verbose=True)
eth_logger = get_logger('okx_collector_eth_usdt', verbose=True)

btc_collector = OKXCollector('BTC-USDT', logger=btc_logger)
eth_collector = OKXCollector('ETH-USDT', logger=eth_logger)

# Add collectors to manager
manager.add_collector(btc_collector)
manager.add_collector(eth_collector)

# Result:
# - Manager logs to: logs/collector_manager/YYYY-MM-DD.txt
# - BTC collector logs to: logs/okx_collector_btc_usdt/YYYY-MM-DD.txt
# - ETH collector logs to: logs/okx_collector_eth_usdt/YYYY-MM-DD.txt
# - All child components of each collector log to their parent's file
```

## Implementation Details

### Base Classes

All base classes support conditional logging:

```python
class BaseDataCollector:
    def __init__(self, ..., logger=None, log_errors_only=False):
        self.logger = logger
        self.log_errors_only = log_errors_only
        
    def _log_debug(self, message: str) -> None:
        if self.logger and not self.log_errors_only:
            self.logger.debug(message)
    
    def _log_error(self, message: str, exc_info: bool = False) -> None:
        if self.logger:
            self.logger.error(message, exc_info=exc_info)
```

### Child Component Pattern

Child components receive logger from parent:

```python
class OKXCollector(BaseDataCollector):
    def __init__(self, symbol: str, logger=None):
        super().__init__(..., logger=logger)
        
        # Pass logger to child components
        self._data_processor = OKXDataProcessor(
            symbol, 
            logger=self.logger  # Pass parent's logger
        )
```

### Conditional Logging Helpers

All components use helper methods for conditional logging:

```python
def _log_debug(self, message: str) -> None:
    """Log debug message if logger is available and not in errors-only mode."""
    if self.logger and not self.log_errors_only:
        self.logger.debug(message)

def _log_info(self, message: str) -> None:
    """Log info message if logger is available and not in errors-only mode."""
    if self.logger and not self.log_errors_only:
        self.logger.info(message)

def _log_warning(self, message: str) -> None:
    """Log warning message if logger is available and not in errors-only mode."""
    if self.logger and not self.log_errors_only:
        self.logger.warning(message)

def _log_error(self, message: str, exc_info: bool = False) -> None:
    """Log error message if logger is available (always logs errors)."""
    if self.logger:
        self.logger.error(message, exc_info=exc_info)

def _log_critical(self, message: str, exc_info: bool = False) -> None:
    """Log critical message if logger is available (always logs critical)."""
    if self.logger:
        self.logger.critical(message, exc_info=exc_info)
```

## Log File Structure

```
logs/
├── collector_manager/
│   └── 2024-01-15.txt
├── okx_collector_btc_usdt/
│   └── 2024-01-15.txt
├── okx_collector_eth_usdt/
│   └── 2024-01-15.txt
└── production_manager/
    └── 2024-01-15.txt
```

## Configuration Options

### Logger Parameters

- `logger`: Logger instance or None
- `log_errors_only`: Boolean flag for error-only mode
- `verbose`: Console output (when creating new loggers)
- `clean_old_logs`: Automatic cleanup of old log files
- `max_log_files`: Maximum number of log files to keep

### Environment Variables

```bash
# Enable verbose console logging
VERBOSE_LOGGING=true

# Enable console output
LOG_TO_CONSOLE=true
```

## Best Practices

### 1. Component Design
- Always accept `logger=None` parameter in constructors
- Pass logger to all child components
- Use conditional logging helper methods
- Never assume logger is available

### 2. Error Handling
- Always log errors regardless of `log_errors_only` setting
- Use appropriate log levels
- Include context in error messages

### 3. Performance
- Conditional logging has minimal performance impact
- Logger checks are fast boolean operations
- No string formatting when logging is disabled

### 4. Testing
- Test components with and without loggers
- Verify error-only mode works correctly
- Check that child components receive loggers properly

## Migration Guide

### Updating Existing Components

1. **Add logger parameter to constructor**:
```python
def __init__(self, ..., logger=None, log_errors_only=False):
```

2. **Add conditional logging helpers**:
```python
def _log_debug(self, message: str) -> None:
    if self.logger and not self.log_errors_only:
        self.logger.debug(message)
```

3. **Update all logging calls**:
```python
# Before
self.logger.info("Message")

# After
self._log_info("Message")
```

4. **Pass logger to child components**:
```python
child = ChildComponent(logger=self.logger)
```

### Testing Changes

```python
# Test without logger
component = MyComponent(logger=None)
# Should work without errors, no logging

# Test with logger
logger = get_logger('test_component')
component = MyComponent(logger=logger)
# Should log normally

# Test error-only mode
component = MyComponent(logger=logger, log_errors_only=True)
# Should only log errors
```

This conditional logging system provides maximum flexibility while maintaining clean, maintainable code that works in all scenarios.