Enhance logging capabilities across data collection components
- Added optional logger parameter to various classes including `BaseDataCollector`, `CollectorManager`, `RealTimeCandleProcessor`, and `BatchCandleProcessor` to support conditional logging. - Implemented error-only logging mode, allowing components to log only error and critical messages when specified. - Updated logging calls to utilize new helper methods for improved readability and maintainability. - Enhanced documentation to include details on the new logging system and its usage across components. - Ensured that child components inherit the logger from their parent components for consistent logging behavior.
This commit is contained in:
Vasily.onl
@@ -1,6 +1,6 @@
|
||||
# Unified Logging System
|
||||
|
||||
The TCP Dashboard project uses a unified logging system that provides consistent, centralized logging across all components.
|
||||
The TCP Dashboard project uses a unified logging system that provides consistent, centralized logging across all components with advanced conditional logging capabilities.
|
||||
|
||||
## Features
|
||||
|
||||
@@ -11,6 +11,315 @@ The TCP Dashboard project uses a unified logging system that provides consistent
|
||||
- **Verbose console logging**: Configurable console output with proper log level handling
|
||||
- **Automatic log cleanup**: Built-in functionality to remove old log files automatically
|
||||
- **Error handling**: Graceful fallback to console logging if file logging fails
|
||||
- **Conditional logging**: Components can operate with or without loggers
|
||||
- **Error-only logging**: Option to log only error-level messages
|
||||
- **Hierarchical logging**: Parent components can pass loggers to children
|
||||
- **Logger inheritance**: Consistent logging across component hierarchies
|
||||
|
||||
## Conditional Logging System
|
||||
|
||||
The TCP Dashboard implements a sophisticated conditional logging system that allows components to work with or without loggers, providing maximum flexibility for different deployment scenarios.
|
||||
|
||||
### Key Concepts
|
||||
|
||||
1. **Optional Logging**: Components accept `logger=None` and function normally without logging
|
||||
2. **Error-Only Mode**: Components can log only error-level messages with `log_errors_only=True`
|
||||
3. **Logger Inheritance**: Parent components pass their logger to child components
|
||||
4. **Hierarchical Structure**: Log files are organized by component hierarchy
|
||||
|
||||
### Usage Patterns
|
||||
|
||||
#### 1. No Logging
|
||||
```python
|
||||
from data.collector_manager import CollectorManager
|
||||
from data.exchanges.okx.collector import OKXCollector
|
||||
|
||||
# Components work without any logging
|
||||
manager = CollectorManager(logger=None)
|
||||
collector = OKXCollector("BTC-USDT", logger=None)
|
||||
|
||||
# No log files created, no console output
|
||||
# Components function normally without exceptions
|
||||
```
|
||||
|
||||
#### 2. Normal Logging
|
||||
```python
|
||||
from utils.logger import get_logger
|
||||
from data.collector_manager import CollectorManager
|
||||
|
||||
# Create logger for the manager
|
||||
logger = get_logger('production_manager')
|
||||
|
||||
# Manager logs all activities
|
||||
manager = CollectorManager(logger=logger)
|
||||
|
||||
# Child components inherit the logger
|
||||
collector = manager.add_okx_collector("BTC-USDT") # Uses manager's logger
|
||||
```
|
||||
|
||||
#### 3. Error-Only Logging
|
||||
```python
|
||||
from utils.logger import get_logger
|
||||
from data.exchanges.okx.collector import OKXCollector
|
||||
|
||||
# Create logger but only log errors
|
||||
logger = get_logger('critical_only')
|
||||
|
||||
# Only error and critical messages are logged
|
||||
collector = OKXCollector(
|
||||
"BTC-USDT",
|
||||
logger=logger,
|
||||
log_errors_only=True
|
||||
)
|
||||
|
||||
# Debug, info, warning messages are suppressed
|
||||
# Error and critical messages are always logged
|
||||
```
|
||||
|
||||
#### 4. Hierarchical Logging
|
||||
```python
|
||||
from utils.logger import get_logger
|
||||
from data.collector_manager import CollectorManager
|
||||
|
||||
# Top-level application logger
|
||||
app_logger = get_logger('tcp_dashboard')
|
||||
|
||||
# Production manager with its own logger
|
||||
prod_logger = get_logger('production_manager')
|
||||
manager = CollectorManager(logger=prod_logger)
|
||||
|
||||
# Individual collectors with specific loggers
|
||||
btc_logger = get_logger('btc_collector')
|
||||
btc_collector = OKXCollector("BTC-USDT", logger=btc_logger)
|
||||
|
||||
eth_collector = OKXCollector("ETH-USDT", logger=None) # No logging
|
||||
|
||||
# Results in organized log structure:
|
||||
# logs/tcp_dashboard/
|
||||
# logs/production_manager/
|
||||
# logs/btc_collector/
|
||||
# (no logs for ETH collector)
|
||||
```
|
||||
|
||||
#### 5. Mixed Configuration
|
||||
```python
|
||||
from utils.logger import get_logger
|
||||
from data.collector_manager import CollectorManager
|
||||
|
||||
# System logger for normal operations
|
||||
system_logger = get_logger('system')
|
||||
|
||||
# Critical logger for error-only components
|
||||
critical_logger = get_logger('critical_only')
|
||||
|
||||
manager = CollectorManager(logger=system_logger)
|
||||
|
||||
# Different logging strategies for different collectors
|
||||
btc_collector = OKXCollector("BTC-USDT", logger=system_logger) # Full logging
|
||||
eth_collector = OKXCollector("ETH-USDT", logger=critical_logger, log_errors_only=True) # Errors only
|
||||
ada_collector = OKXCollector("ADA-USDT", logger=None) # No logging
|
||||
|
||||
manager.add_collector(btc_collector)
|
||||
manager.add_collector(eth_collector)
|
||||
manager.add_collector(ada_collector)
|
||||
```
|
||||
|
||||
### Implementation Details
|
||||
|
||||
#### Component Constructor Pattern
|
||||
All major components follow this pattern:
|
||||
```python
|
||||
class ComponentExample:
|
||||
def __init__(self, logger=None, log_errors_only=False):
|
||||
self.logger = logger
|
||||
self.log_errors_only = log_errors_only
|
||||
|
||||
# Conditional logging helpers
|
||||
self._log_debug = self._create_conditional_logger('debug')
|
||||
self._log_info = self._create_conditional_logger('info')
|
||||
self._log_warning = self._create_conditional_logger('warning')
|
||||
self._log_error = self._create_conditional_logger('error')
|
||||
self._log_critical = self._create_conditional_logger('critical')
|
||||
|
||||
def _create_conditional_logger(self, level):
|
||||
"""Create conditional logging function based on configuration."""
|
||||
if not self.logger:
|
||||
return lambda msg: None # No-op if no logger
|
||||
|
||||
log_func = getattr(self.logger, level)
|
||||
|
||||
if level in ['debug', 'info', 'warning'] and self.log_errors_only:
|
||||
return lambda msg: None # Suppress non-error messages
|
||||
|
||||
return log_func # Normal logging
|
||||
```
|
||||
|
||||
#### Supported Components
|
||||
|
||||
The following components support conditional logging:
|
||||
|
||||
1. **BaseDataCollector** (`data/base_collector.py`)
|
||||
- Parameters: `logger=None, log_errors_only=False`
|
||||
- Conditional logging for all collector operations
|
||||
|
||||
2. **CollectorManager** (`data/collector_manager.py`)
|
||||
- Parameters: `logger=None, log_errors_only=False`
|
||||
- Manages multiple collectors with consistent logging
|
||||
|
||||
3. **OKXCollector** (`data/exchanges/okx/collector.py`)
|
||||
- Parameters: `logger=None, log_errors_only=False`
|
||||
- Exchange-specific data collection with conditional logging
|
||||
|
||||
4. **BaseDataValidator** (`data/common/validation.py`)
|
||||
- Parameters: `logger=None`
|
||||
- Data validation with optional logging
|
||||
|
||||
5. **OKXDataTransformer** (`data/exchanges/okx/data_processor.py`)
|
||||
- Parameters: `logger=None`
|
||||
- Data processing with conditional logging
|
||||
|
||||
### Best Practices for Conditional Logging
|
||||
|
||||
#### 1. Logger Inheritance
|
||||
```python
|
||||
# Parent component creates logger
|
||||
parent_logger = get_logger('parent_system')
|
||||
parent = ParentComponent(logger=parent_logger)
|
||||
|
||||
# Pass logger to children for consistent hierarchy
|
||||
child1 = ChildComponent(logger=parent_logger)
|
||||
child2 = ChildComponent(logger=parent_logger, log_errors_only=True)
|
||||
child3 = ChildComponent(logger=None) # No logging
|
||||
```
|
||||
|
||||
#### 2. Environment-Based Configuration
|
||||
```python
|
||||
import os
|
||||
from utils.logger import get_logger
|
||||
|
||||
def create_system_logger():
|
||||
"""Create logger based on environment."""
|
||||
env = os.getenv('ENVIRONMENT', 'development')
|
||||
|
||||
if env == 'production':
|
||||
return get_logger('production_system', log_level='INFO', verbose=False)
|
||||
elif env == 'testing':
|
||||
return None # No logging during tests
|
||||
else:
|
||||
return get_logger('dev_system', log_level='DEBUG', verbose=True)
|
||||
|
||||
# Use in components
|
||||
system_logger = create_system_logger()
|
||||
manager = CollectorManager(logger=system_logger)
|
||||
```
|
||||
|
||||
#### 3. Conditional Error-Only Mode
|
||||
```python
|
||||
def create_collector_with_logging_strategy(symbol, strategy='normal'):
|
||||
"""Create collector with different logging strategies."""
|
||||
base_logger = get_logger(f'collector_{symbol.lower().replace("-", "_")}')
|
||||
|
||||
if strategy == 'silent':
|
||||
return OKXCollector(symbol, logger=None)
|
||||
elif strategy == 'errors_only':
|
||||
return OKXCollector(symbol, logger=base_logger, log_errors_only=True)
|
||||
else:
|
||||
return OKXCollector(symbol, logger=base_logger)
|
||||
|
||||
# Usage
|
||||
btc_collector = create_collector_with_logging_strategy('BTC-USDT', 'normal')
|
||||
eth_collector = create_collector_with_logging_strategy('ETH-USDT', 'errors_only')
|
||||
ada_collector = create_collector_with_logging_strategy('ADA-USDT', 'silent')
|
||||
```
|
||||
|
||||
#### 4. Performance Optimization
|
||||
```python
|
||||
class OptimizedComponent:
|
||||
def __init__(self, logger=None, log_errors_only=False):
|
||||
self.logger = logger
|
||||
self.log_errors_only = log_errors_only
|
||||
|
||||
# Pre-compute logging capabilities for performance
|
||||
self.can_log_debug = logger and not log_errors_only
|
||||
self.can_log_info = logger and not log_errors_only
|
||||
self.can_log_warning = logger and not log_errors_only
|
||||
self.can_log_error = logger is not None
|
||||
self.can_log_critical = logger is not None
|
||||
|
||||
def process_data(self, data):
|
||||
if self.can_log_debug:
|
||||
self.logger.debug(f"Processing {len(data)} records")
|
||||
|
||||
# ... processing logic ...
|
||||
|
||||
if self.can_log_info:
|
||||
self.logger.info("Data processing completed")
|
||||
```
|
||||
|
||||
### Migration Guide
|
||||
|
||||
#### From Standard Logging
|
||||
```python
|
||||
# Old approach
|
||||
import logging
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
class OldComponent:
|
||||
def __init__(self):
|
||||
self.logger = logger
|
||||
|
||||
# New conditional approach
|
||||
from utils.logger import get_logger
|
||||
|
||||
class NewComponent:
|
||||
def __init__(self, logger=None, log_errors_only=False):
|
||||
self.logger = logger
|
||||
self.log_errors_only = log_errors_only
|
||||
|
||||
# Add conditional logging helpers
|
||||
self._setup_conditional_logging()
|
||||
```
|
||||
|
||||
#### Gradual Adoption
|
||||
1. **Phase 1**: Add optional logger parameters to new components
|
||||
2. **Phase 2**: Update existing components to support conditional logging
|
||||
3. **Phase 3**: Implement hierarchical logging structure
|
||||
4. **Phase 4**: Add error-only logging mode
|
||||
|
||||
### Testing Conditional Logging
|
||||
|
||||
#### Test Script Example
|
||||
```python
|
||||
# test_conditional_logging.py
|
||||
from utils.logger import get_logger
|
||||
from data.collector_manager import CollectorManager
|
||||
from data.exchanges.okx.collector import OKXCollector
|
||||
|
||||
def test_no_logging():
|
||||
"""Test components work without loggers."""
|
||||
manager = CollectorManager(logger=None)
|
||||
collector = OKXCollector("BTC-USDT", logger=None)
|
||||
print("✓ No logging test passed")
|
||||
|
||||
def test_with_logging():
|
||||
"""Test components work with loggers."""
|
||||
logger = get_logger('test_system')
|
||||
manager = CollectorManager(logger=logger)
|
||||
collector = OKXCollector("BTC-USDT", logger=logger)
|
||||
print("✓ With logging test passed")
|
||||
|
||||
def test_error_only():
|
||||
"""Test error-only logging mode."""
|
||||
logger = get_logger('test_errors')
|
||||
collector = OKXCollector("BTC-USDT", logger=logger, log_errors_only=True)
|
||||
print("✓ Error-only logging test passed")
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_no_logging()
|
||||
test_with_logging()
|
||||
test_error_only()
|
||||
print("✅ All conditional logging tests passed!")
|
||||
```
|
||||
|
||||
## Log Format
|
||||
|
||||
|
||||
Reference in New Issue
Block a user