vasily/TCPDashboard
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Enhance logging capabilities across data collection components

Browse Source 
- Added optional logger parameter to various classes including `BaseDataCollector`, `CollectorManager`, `RealTimeCandleProcessor`, and `BatchCandleProcessor` to support conditional logging.
- Implemented error-only logging mode, allowing components to log only error and critical messages when specified.
- Updated logging calls to utilize new helper methods for improved readability and maintainability.
- Enhanced documentation to include details on the new logging system and its usage across components.
- Ensured that child components inherit the logger from their parent components for consistent logging behavior.
This commit is contained in:
Vasily.onl
2025-06-01 14:42:29 +08:00
parent 0697be75da
commit bc13cfcbe0
11 changed files with 1179 additions and 350 deletions
Download Patch File Download Diff File Expand all files Collapse all files

311
docs/components/logging.md
View File

@@ -1,6 +1,6 @@
# Unified Logging System
The TCP Dashboard project uses a unified logging system that provides consistent, centralized logging across all components.
The TCP Dashboard project uses a unified logging system that provides consistent, centralized logging across all components with advanced conditional logging capabilities.
## Features
@@ -11,6 +11,315 @@ The TCP Dashboard project uses a unified logging system that provides consistent
- **Verbose console logging**: Configurable console output with proper log level handling
- **Automatic log cleanup**: Built-in functionality to remove old log files automatically
- **Error handling**: Graceful fallback to console logging if file logging fails
- **Conditional logging**: Components can operate with or without loggers
- **Error-only logging**: Option to log only error-level messages
- **Hierarchical logging**: Parent components can pass loggers to children
- **Logger inheritance**: Consistent logging across component hierarchies
## Conditional Logging System
The TCP Dashboard implements a sophisticated conditional logging system that allows components to work with or without loggers, providing maximum flexibility for different deployment scenarios.
### Key Concepts
1. **Optional Logging**: Components accept `logger=None` and function normally without logging
2. **Error-Only Mode**: Components can log only error-level messages with `log_errors_only=True`
3. **Logger Inheritance**: Parent components pass their logger to child components
4. **Hierarchical Structure**: Log files are organized by component hierarchy
### Usage Patterns
#### 1. No Logging
```python
from data.collector_manager import CollectorManager
from data.exchanges.okx.collector import OKXCollector
# Components work without any logging
manager = CollectorManager(logger=None)
collector = OKXCollector("BTC-USDT", logger=None)
# No log files created, no console output
# Components function normally without exceptions
```
#### 2. Normal Logging
```python
from utils.logger import get_logger
from data.collector_manager import CollectorManager
# Create logger for the manager
logger = get_logger('production_manager')
# Manager logs all activities
manager = CollectorManager(logger=logger)
# Child components inherit the logger
collector = manager.add_okx_collector("BTC-USDT") # Uses manager's logger
```
#### 3. Error-Only Logging
```python
from utils.logger import get_logger
from data.exchanges.okx.collector import OKXCollector
# Create logger but only log errors
logger = get_logger('critical_only')
# Only error and critical messages are logged
collector = OKXCollector(
"BTC-USDT",
logger=logger,
log_errors_only=True
)
# Debug, info, warning messages are suppressed
# Error and critical messages are always logged
```
#### 4. Hierarchical Logging
```python
from utils.logger import get_logger
from data.collector_manager import CollectorManager
# Top-level application logger
app_logger = get_logger('tcp_dashboard')
# Production manager with its own logger
prod_logger = get_logger('production_manager')
manager = CollectorManager(logger=prod_logger)
# Individual collectors with specific loggers
btc_logger = get_logger('btc_collector')
btc_collector = OKXCollector("BTC-USDT", logger=btc_logger)
eth_collector = OKXCollector("ETH-USDT", logger=None) # No logging
# Results in organized log structure:
# logs/tcp_dashboard/
# logs/production_manager/
# logs/btc_collector/
# (no logs for ETH collector)
```
#### 5. Mixed Configuration
```python
from utils.logger import get_logger
from data.collector_manager import CollectorManager
# System logger for normal operations
system_logger = get_logger('system')
# Critical logger for error-only components
critical_logger = get_logger('critical_only')
manager = CollectorManager(logger=system_logger)
# Different logging strategies for different collectors
btc_collector = OKXCollector("BTC-USDT", logger=system_logger) # Full logging
eth_collector = OKXCollector("ETH-USDT", logger=critical_logger, log_errors_only=True) # Errors only
ada_collector = OKXCollector("ADA-USDT", logger=None) # No logging
manager.add_collector(btc_collector)
manager.add_collector(eth_collector)
manager.add_collector(ada_collector)
```
### Implementation Details
#### Component Constructor Pattern
All major components follow this pattern:
```python
class ComponentExample:
def __init__(self, logger=None, log_errors_only=False):
self.logger = logger
self.log_errors_only = log_errors_only
# Conditional logging helpers
self._log_debug = self._create_conditional_logger('debug')
self._log_info = self._create_conditional_logger('info')
self._log_warning = self._create_conditional_logger('warning')
self._log_error = self._create_conditional_logger('error')
self._log_critical = self._create_conditional_logger('critical')
def _create_conditional_logger(self, level):
"""Create conditional logging function based on configuration."""
if not self.logger:
return lambda msg: None # No-op if no logger
log_func = getattr(self.logger, level)
if level in ['debug', 'info', 'warning'] and self.log_errors_only:
return lambda msg: None # Suppress non-error messages
return log_func # Normal logging
```
#### Supported Components
The following components support conditional logging:
1. **BaseDataCollector** (`data/base_collector.py`)
- Parameters: `logger=None, log_errors_only=False`
- Conditional logging for all collector operations
2. **CollectorManager** (`data/collector_manager.py`)
- Parameters: `logger=None, log_errors_only=False`
- Manages multiple collectors with consistent logging
3. **OKXCollector** (`data/exchanges/okx/collector.py`)
- Parameters: `logger=None, log_errors_only=False`
- Exchange-specific data collection with conditional logging
4. **BaseDataValidator** (`data/common/validation.py`)
- Parameters: `logger=None`
- Data validation with optional logging
5. **OKXDataTransformer** (`data/exchanges/okx/data_processor.py`)
- Parameters: `logger=None`
- Data processing with conditional logging
### Best Practices for Conditional Logging
#### 1. Logger Inheritance
```python
# Parent component creates logger
parent_logger = get_logger('parent_system')
parent = ParentComponent(logger=parent_logger)
# Pass logger to children for consistent hierarchy
child1 = ChildComponent(logger=parent_logger)
child2 = ChildComponent(logger=parent_logger, log_errors_only=True)
child3 = ChildComponent(logger=None) # No logging
```
#### 2. Environment-Based Configuration
```python
import os
from utils.logger import get_logger
def create_system_logger():
"""Create logger based on environment."""
env = os.getenv('ENVIRONMENT', 'development')
if env == 'production':
return get_logger('production_system', log_level='INFO', verbose=False)
elif env == 'testing':
return None # No logging during tests
else:
return get_logger('dev_system', log_level='DEBUG', verbose=True)
# Use in components
system_logger = create_system_logger()
manager = CollectorManager(logger=system_logger)
```
#### 3. Conditional Error-Only Mode
```python
def create_collector_with_logging_strategy(symbol, strategy='normal'):
"""Create collector with different logging strategies."""
base_logger = get_logger(f'collector_{symbol.lower().replace("-", "_")}')
if strategy == 'silent':
return OKXCollector(symbol, logger=None)
elif strategy == 'errors_only':
return OKXCollector(symbol, logger=base_logger, log_errors_only=True)
else:
return OKXCollector(symbol, logger=base_logger)
# Usage
btc_collector = create_collector_with_logging_strategy('BTC-USDT', 'normal')
eth_collector = create_collector_with_logging_strategy('ETH-USDT', 'errors_only')
ada_collector = create_collector_with_logging_strategy('ADA-USDT', 'silent')
```
#### 4. Performance Optimization
```python
class OptimizedComponent:
def __init__(self, logger=None, log_errors_only=False):
self.logger = logger
self.log_errors_only = log_errors_only
# Pre-compute logging capabilities for performance
self.can_log_debug = logger and not log_errors_only
self.can_log_info = logger and not log_errors_only
self.can_log_warning = logger and not log_errors_only
self.can_log_error = logger is not None
self.can_log_critical = logger is not None
def process_data(self, data):
if self.can_log_debug:
self.logger.debug(f"Processing {len(data)} records")
# ... processing logic ...
if self.can_log_info:
self.logger.info("Data processing completed")
```
### Migration Guide
#### From Standard Logging
```python
# Old approach
import logging
logger = logging.getLogger(__name__)
class OldComponent:
def __init__(self):
self.logger = logger
# New conditional approach
from utils.logger import get_logger
class NewComponent:
def __init__(self, logger=None, log_errors_only=False):
self.logger = logger
self.log_errors_only = log_errors_only
# Add conditional logging helpers
self._setup_conditional_logging()
```
#### Gradual Adoption
1. **Phase 1**: Add optional logger parameters to new components
2. **Phase 2**: Update existing components to support conditional logging
3. **Phase 3**: Implement hierarchical logging structure
4. **Phase 4**: Add error-only logging mode
### Testing Conditional Logging
#### Test Script Example
```python
# test_conditional_logging.py
from utils.logger import get_logger
from data.collector_manager import CollectorManager
from data.exchanges.okx.collector import OKXCollector
def test_no_logging():
"""Test components work without loggers."""
manager = CollectorManager(logger=None)
collector = OKXCollector("BTC-USDT", logger=None)
print("✓ No logging test passed")
def test_with_logging():
"""Test components work with loggers."""
logger = get_logger('test_system')
manager = CollectorManager(logger=logger)
collector = OKXCollector("BTC-USDT", logger=logger)
print("✓ With logging test passed")
def test_error_only():
"""Test error-only logging mode."""
logger = get_logger('test_errors')
collector = OKXCollector("BTC-USDT", logger=logger, log_errors_only=True)
print("✓ Error-only logging test passed")
if __name__ == "__main__":
test_no_logging()
test_with_logging()
test_error_only()
print("✅ All conditional logging tests passed!")
```
## Log Format

292
docs/logging_system.md Normal file
View File

@@ -0,0 +1,292 @@
# Conditional Logging System
## Overview
The TCP Dashboard project implements a sophisticated conditional logging system that provides fine-grained control over logging behavior across all components. This system supports hierarchical logging, conditional logging, and error-only logging modes.
## Key Features
### 1. Conditional Logging
- **No Logger**: If no logger instance is passed to a component's constructor, that component performs no logging operations
- **Logger Provided**: If a logger instance is passed, the component uses it for logging
- **Error-Only Mode**: If `log_errors_only=True` is set, only error and critical level messages are logged
### 2. Logger Inheritance
- Components that receive a logger pass the same logger instance down to child components
- This creates a hierarchical logging structure that follows the component hierarchy
### 3. Hierarchical File Organization
- Log files are organized based on component hierarchy
- Each major component gets its own log directory
- Child components log to their parent's log file
## Component Hierarchy
```
Top-level Application (individual logger)
├── ProductionManager (individual logger)
│ ├── DataSaver (receives logger from ProductionManager)
│ ├── DataValidator (receives logger from ProductionManager)
│ ├── DatabaseConnection (receives logger from ProductionManager)
│ └── CollectorManager (individual logger)
│ ├── OKX collector BTC-USD (individual logger)
│ │ ├── DataAggregator (receives logger from OKX collector)
│ │ ├── DataTransformer (receives logger from OKX collector)
│ │ └── DataProcessor (receives logger from OKX collector)
│ └── Another collector...
```
## Usage Examples
### Basic Usage
```python
from utils.logger import get_logger
from data.exchanges.okx.collector import OKXCollector
# Create a logger for the collector
collector_logger = get_logger('okx_collector_btc_usdt', verbose=True)
# Create collector with logger - all child components will use this logger
collector = OKXCollector(
symbol='BTC-USDT',
logger=collector_logger
)
# Child components (data processor, validator, transformer) will automatically
# receive and use the same logger instance
```
### No Logging Mode
```python
# Create collector without logger - no logging will be performed
collector = OKXCollector(
symbol='BTC-USDT',
logger=None # or simply omit the parameter
)
# No log files will be created, no console output
```
### Error-Only Logging Mode
```python
from utils.logger import get_logger
from data.collector_manager import CollectorManager
# Create logger for manager
manager_logger = get_logger('collector_manager', verbose=True)
# Create manager with error-only logging
manager = CollectorManager(
manager_name="production_manager",
logger=manager_logger,
log_errors_only=True # Only errors and critical messages will be logged
)
# Manager will only log errors, but child collectors can have their own loggers
```
### Hierarchical Logging Setup
```python
from utils.logger import get_logger
from data.collector_manager import CollectorManager
from data.exchanges.okx.collector import OKXCollector
# Create manager with its own logger
manager_logger = get_logger('collector_manager', verbose=True)
manager = CollectorManager(logger=manager_logger)
# Create individual collectors with their own loggers
btc_logger = get_logger('okx_collector_btc_usdt', verbose=True)
eth_logger = get_logger('okx_collector_eth_usdt', verbose=True)
btc_collector = OKXCollector('BTC-USDT', logger=btc_logger)
eth_collector = OKXCollector('ETH-USDT', logger=eth_logger)
# Add collectors to manager
manager.add_collector(btc_collector)
manager.add_collector(eth_collector)
# Result:
# - Manager logs to: logs/collector_manager/YYYY-MM-DD.txt
# - BTC collector logs to: logs/okx_collector_btc_usdt/YYYY-MM-DD.txt
# - ETH collector logs to: logs/okx_collector_eth_usdt/YYYY-MM-DD.txt
# - All child components of each collector log to their parent's file
```
## Implementation Details
### Base Classes
All base classes support conditional logging:
```python
class BaseDataCollector:
def __init__(self, ..., logger=None, log_errors_only=False):
self.logger = logger
self.log_errors_only = log_errors_only
def _log_debug(self, message: str) -> None:
if self.logger and not self.log_errors_only:
self.logger.debug(message)
def _log_error(self, message: str, exc_info: bool = False) -> None:
if self.logger:
self.logger.error(message, exc_info=exc_info)
```
### Child Component Pattern
Child components receive logger from parent:
```python
class OKXCollector(BaseDataCollector):
def __init__(self, symbol: str, logger=None):
super().__init__(..., logger=logger)
# Pass logger to child components
self._data_processor = OKXDataProcessor(
symbol,
logger=self.logger # Pass parent's logger
)
```
### Conditional Logging Helpers
All components use helper methods for conditional logging:
```python
def _log_debug(self, message: str) -> None:
"""Log debug message if logger is available and not in errors-only mode."""
if self.logger and not self.log_errors_only:
self.logger.debug(message)
def _log_info(self, message: str) -> None:
"""Log info message if logger is available and not in errors-only mode."""
if self.logger and not self.log_errors_only:
self.logger.info(message)
def _log_warning(self, message: str) -> None:
"""Log warning message if logger is available and not in errors-only mode."""
if self.logger and not self.log_errors_only:
self.logger.warning(message)
def _log_error(self, message: str, exc_info: bool = False) -> None:
"""Log error message if logger is available (always logs errors)."""
if self.logger:
self.logger.error(message, exc_info=exc_info)
def _log_critical(self, message: str, exc_info: bool = False) -> None:
"""Log critical message if logger is available (always logs critical)."""
if self.logger:
self.logger.critical(message, exc_info=exc_info)
```
## Log File Structure
```
logs/
├── collector_manager/
│ └── 2024-01-15.txt
├── okx_collector_btc_usdt/
│ └── 2024-01-15.txt
├── okx_collector_eth_usdt/
│ └── 2024-01-15.txt
└── production_manager/
└── 2024-01-15.txt
```
## Configuration Options
### Logger Parameters
- `logger`: Logger instance or None
- `log_errors_only`: Boolean flag for error-only mode
- `verbose`: Console output (when creating new loggers)
- `clean_old_logs`: Automatic cleanup of old log files
- `max_log_files`: Maximum number of log files to keep
### Environment Variables
```bash
# Enable verbose console logging
VERBOSE_LOGGING=true
# Enable console output
LOG_TO_CONSOLE=true
```
## Best Practices
### 1. Component Design
- Always accept `logger=None` parameter in constructors
- Pass logger to all child components
- Use conditional logging helper methods
- Never assume logger is available
### 2. Error Handling
- Always log errors regardless of `log_errors_only` setting
- Use appropriate log levels
- Include context in error messages
### 3. Performance
- Conditional logging has minimal performance impact
- Logger checks are fast boolean operations
- No string formatting when logging is disabled
### 4. Testing
- Test components with and without loggers
- Verify error-only mode works correctly
- Check that child components receive loggers properly
## Migration Guide
### Updating Existing Components
1. **Add logger parameter to constructor**:
```python
def __init__(self, ..., logger=None, log_errors_only=False):
```
2. **Add conditional logging helpers**:
```python
def _log_debug(self, message: str) -> None:
if self.logger and not self.log_errors_only:
self.logger.debug(message)
```
3. **Update all logging calls**:
```python
# Before
self.logger.info("Message")
# After
self._log_info("Message")
```
4. **Pass logger to child components**:
```python
child = ChildComponent(logger=self.logger)
```
### Testing Changes
```python
# Test without logger
component = MyComponent(logger=None)
# Should work without errors, no logging
# Test with logger
logger = get_logger('test_component')
component = MyComponent(logger=logger)
# Should log normally
# Test error-only mode
component = MyComponent(logger=logger, log_errors_only=True)
# Should only log errors
```
This conditional logging system provides maximum flexibility while maintaining clean, maintainable code that works in all scenarios.