vasily/TCPDashboard
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update logging documentation and refactor logger implementation

Browse Source 
- Revised the logging documentation to clarify the unified logging system's features and usage patterns.
- Simplified the logger implementation by removing the custom `DateRotatingFileHandler` and utilizing the standard library's `TimedRotatingFileHandler` for date-based log rotation.
- Enhanced the `get_logger` function to ensure thread-safe logger configuration and prevent duplicate handlers.
- Introduced a new `cleanup_old_logs` function for age-based log cleanup, while retaining the existing count-based cleanup mechanism.
- Improved error handling and logging setup to ensure robust logging behavior across components.

These changes enhance the clarity and maintainability of the logging system, making it easier for developers to implement and utilize logging in their components.
This commit is contained in:
Vasily.onl
2025-06-06 21:02:08 +08:00
parent c1118eaf2b
commit e147aa1873
2 changed files with 200 additions and 545 deletions
Download Patch File Download Diff File Expand all files Collapse all files

343
docs/modules/logging.md
View File

@@ -1,246 +1,33 @@
# Unified Logging System
The TCP Dashboard project uses a unified logging system that provides consistent, centralized logging across all components with advanced conditional logging capabilities.
The TCP Dashboard project uses a unified logging system built on Python's standard `logging` library. It provides consistent, centralized logging across all components.
## Key Features
- **Component-based logging**: Each component (e.g., `bot_manager`, `data_collector`) gets its own dedicated logger and log directory under `logs/`.
- **Centralized control**: `UnifiedLogger` class manages all logger instances, ensuring consistent configuration.
- **Date-based rotation**: Log files are automatically rotated daily (e.g., `2023-11-15.txt`).
- **Unified format**: All log messages follow `[YYYY-MM-DD HH:MM:SS - LEVEL - message]`.
- **Verbose console logging**: Optional verbose console output for real-time monitoring, controlled by environment variables.
- **Automatic cleanup**: Old log files are automatically removed to save disk space.
## Features
- **Component-specific directories**: Each component gets its own log directory
- **Date-based file rotation**: New log files created daily automatically
- **Unified format**: Consistent timestamp and message format across all logs
- **Thread-safe**: Safe for use in multi-threaded applications
- **Verbose console logging**: Configurable console output with proper log level handling
- **Automatic log cleanup**: Built-in functionality to remove old log files automatically
- **Error handling**: Graceful fallback to console logging if file logging fails
- **Conditional logging**: Components can operate with or without loggers
- **Error-only logging**: Option to log only error-level messages
- **Hierarchical logging**: Parent components can pass loggers to children
- **Logger inheritance**: Consistent logging across component hierarchies
## Conditional Logging System
The TCP Dashboard implements a sophisticated conditional logging system that allows components to work with or without loggers, providing maximum flexibility for different deployment scenarios.
### Key Concepts
1. **Optional Logging**: Components accept `logger=None` and function normally without logging
2. **Error-Only Mode**: Components can log only error-level messages with `log_errors_only=True`
3. **Logger Inheritance**: Parent components pass their logger to child components
4. **Hierarchical Structure**: Log files are organized by component hierarchy
### Component Hierarchy
```
Top-level Application (individual logger)
├── ProductionManager (individual logger)
│ ├── DataSaver (receives logger from ProductionManager)
│ ├── DataValidator (receives logger from ProductionManager)
│ ├── DatabaseConnection (receives logger from ProductionManager)
│ └── CollectorManager (individual logger)
│ ├── OKX collector BTC-USD (individual logger)
│ │ ├── DataAggregator (receives logger from OKX collector)
│ │ ├── DataTransformer (receives logger from OKX collector)
│ │ └── DataProcessor (receives logger from OKX collector)
│ └── Another collector...
```
### Usage Patterns
#### 1. No Logging
```python
from data.collector_manager import CollectorManager
from data.exchanges.okx.collector import OKXCollector
# Components work without any logging
manager = CollectorManager(logger=None)
collector = OKXCollector("BTC-USDT", logger=None)
# No log files created, no console output
# Components function normally without exceptions
```
#### 2. Normal Logging
```python
from utils.logger import get_logger
from data.collector_manager import CollectorManager
# Create logger for the manager
logger = get_logger('production_manager')
# Manager logs all activities
manager = CollectorManager(logger=logger)
# Child components inherit the logger
collector = manager.add_okx_collector("BTC-USDT") # Uses manager's logger
```
#### 3. Error-Only Logging
```python
from utils.logger import get_logger
from data.exchanges.okx.collector import OKXCollector
# Create logger but only log errors
logger = get_logger('critical_only')
# Only error and critical messages are logged
collector = OKXCollector(
"BTC-USDT",
logger=logger,
log_errors_only=True
)
# Debug, info, warning messages are suppressed
# Error and critical messages are always logged
```
#### 4. Hierarchical Logging
```python
from utils.logger import get_logger
from data.collector_manager import CollectorManager
# Top-level application logger
app_logger = get_logger('tcp_dashboard')
# Production manager with its own logger
prod_logger = get_logger('production_manager')
manager = CollectorManager(logger=prod_logger)
# Individual collectors with specific loggers
btc_logger = get_logger('btc_collector')
btc_collector = OKXCollector("BTC-USDT", logger=btc_logger)
eth_collector = OKXCollector("ETH-USDT", logger=None) # No logging
# Results in organized log structure:
# logs/tcp_dashboard/
# logs/production_manager/
# logs/btc_collector/
# (no logs for ETH collector)
```
#### 5. Mixed Configuration
```python
from utils.logger import get_logger
from data.collector_manager import CollectorManager
# System logger for normal operations
system_logger = get_logger('system')
# Critical logger for error-only components
critical_logger = get_logger('critical_only')
manager = CollectorManager(logger=system_logger)
# Different logging strategies for different collectors
btc_collector = OKXCollector("BTC-USDT", logger=system_logger) # Full logging
eth_collector = OKXCollector("ETH-USDT", logger=critical_logger, log_errors_only=True) # Errors only
ada_collector = OKXCollector("ADA-USDT", logger=None) # No logging
manager.add_collector(btc_collector)
manager.add_collector(eth_collector)
manager.add_collector(ada_collector)
```
### Implementation Details
#### Component Constructor Pattern
All major components follow this pattern:
```python
class ComponentExample:
def __init__(self, logger=None, log_errors_only=False):
self.logger = logger
self.log_errors_only = log_errors_only
def _log_debug(self, message: str) -> None:
"""Log debug message if logger is available and not in errors-only mode."""
if self.logger and not self.log_errors_only:
self.logger.debug(message)
def _log_info(self, message: str) -> None:
"""Log info message if logger is available and not in errors-only mode."""
if self.logger and not self.log_errors_only:
self.logger.info(message)
def _log_warning(self, message: str) -> None:
"""Log warning message if logger is available and not in errors-only mode."""
if self.logger and not self.log_errors_only:
self.logger.warning(message)
def _log_error(self, message: str, exc_info: bool = False) -> None:
"""Log error message if logger is available (always logs errors)."""
if self.logger:
self.logger.error(message, exc_info=exc_info)
def _log_critical(self, message: str, exc_info: bool = False) -> None:
"""Log critical message if logger is available (always logs critical)."""
if self.logger:
self.logger.critical(message, exc_info=exc_info)
```
#### Child Component Pattern
Child components receive logger from parent:
```python
class OKXCollector(BaseDataCollector):
def __init__(self, symbol: str, logger=None, log_errors_only=False):
super().__init__(..., logger=logger, log_errors_only=log_errors_only)
# Pass logger to child components
self._data_processor = OKXDataProcessor(
symbol,
logger=self.logger # Pass parent's logger
)
self._data_validator = DataValidator(logger=self.logger)
self._data_transformer = DataTransformer(logger=self.logger)
```
#### Supported Components
The following components support conditional logging:
1. **BaseDataCollector** (`data/base_collector.py`)
- Parameters: `logger=None, log_errors_only=False`
- Conditional logging for all collector operations
2. **CollectorManager** (`data/collector_manager.py`)
- Parameters: `logger=None, log_errors_only=False`
- Manages multiple collectors with consistent logging
3. **OKXCollector** (`data/exchanges/okx/collector.py`)
- Parameters: `logger=None, log_errors_only=False`
- Exchange-specific data collection with conditional logging
4. **BaseDataValidator** (`data/common/validation.py`)
- Parameters: `logger=None`
- Data validation with optional logging
5. **OKXDataTransformer** (`data/exchanges/okx/data_processor.py`)
- Parameters: `logger=None`
- Data processing with conditional logging
- **Component-based Logging**: Each component (e.g., `bot_manager`, `data_collector`) gets its own dedicated logger, with logs organized into separate directories under `logs/`.
- **Standardized & Simple**: Relies on standard Python `logging` handlers, making it robust and easy to maintain.
- **Date-based Rotation**: Log files are automatically rotated daily at midnight by `TimedRotatingFileHandler`.
- **Automatic Cleanup**: Log file retention is managed automatically based on the number of backup files to keep (`backupCount`), preventing excessive disk usage.
- **Unified Format**: All log messages follow a consistent format: `[YYYY-MM-DD HH:MM:SS - LEVEL - message]`.
- **Configurable Console Output**: Optional verbose console output for real-time monitoring, configurable via function arguments or environment variables.
## Usage
### Getting a Logger
The primary way to get a logger is via the `get_logger` function. It is thread-safe and ensures that loggers are configured only once.
```python
from utils.logger import get_logger
# Get logger for bot manager
# Get a logger for the bot manager component
# This will create a file logger and, if verbose=True, a console logger.
logger = get_logger('bot_manager', verbose=True)
logger.info("Bot started successfully")
logger.debug("Connecting to database...")
logger.warning("API response time is high")
logger.error("Failed to execute trade", extra={'trade_id': 12345})
logger.error("Failed to execute trade", exc_info=True)
```
### Configuration
@@ -249,41 +36,97 @@ The `get_logger` function accepts the following parameters:
| Parameter | Type | Default | Description |
|-------------------|---------------------|---------|-----------------------------------------------------------------------------|
| `component_name` | `str` | - | Name of the component (e.g., `bot_manager`, `data_collector`) |
| `log_level` | `str` | `INFO` | Logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL) |
| `verbose` | `Optional[bool]` | `None` | Enable console logging. If `None`, uses `VERBOSE_LOGGING` from `.env` |
| `clean_old_logs` | `bool` | `True` | Automatically clean old log files when creating new ones |
| `max_log_files` | `int` | `30` | Maximum number of log files to keep per component |
| `component_name` | `str` | - | Name of the component (e.g., `bot_manager`). Used for the logger name and directory. |
| `log_level` | `str` | `INFO` | The minimum logging level to be processed (DEBUG, INFO, WARNING, ERROR, CRITICAL). |
| `verbose` | `Optional[bool]` | `None` | If `True`, enables console logging. If `None`, uses `VERBOSE_LOGGING` or `LOG_TO_CONSOLE` from environment variables. |
| `max_log_files` | `int` | `30` | The maximum number of backup log files to keep. The core of the log cleanup mechanism. |
| `clean_old_logs` | `bool` | `True` | **Deprecated**. Kept for backward compatibility but has no effect. Cleanup is controlled by `max_log_files`. |
## Log Cleanup
For centralized control, you can use environment variables:
- `VERBOSE_LOGGING`: Set to `true` to enable console logging for all loggers.
- `LOG_TO_CONSOLE`: An alias for `VERBOSE_LOGGING`.
Log cleanup is now based on the number of files, not age.
- **Enabled by default**: `clean_old_logs=True`
- **Default retention**: Keeps the most recent 30 log files (`max_log_files=30`)
### Log File Structure
## Centralized Control
For consistent logging behavior across the application, it is recommended to use environment variables in an `.env` file instead of passing parameters to `get_logger`.
- `LOG_LEVEL`: "INFO", "DEBUG", etc.
- `VERBOSE_LOGGING`: "true" or "false"
- `CLEAN_OLD_LOGS`: "true" or "false"
- `MAX_LOG_FILES`: e.g., "15"
## File Structure
The logger creates a directory for each component inside `logs/`. The main log file is named `component_name.log`. When rotated, old logs are renamed with a date suffix.
```
logs/
├── bot_manager/
│ ├── 2023-11-14.txt
│ └── 2023-11-15.txt
│ ├── bot_manager.log (current log file)
│ └── bot_manager.log.2023-11-15
├── data_collector/
│ ├── 2023-11-14.txt
│ └── 2023-11-15.txt
└── default_logger/
└── 2023-11-15.txt
│ ├── data_collector.log
│ └── data_collector.log.2023-11-15
└── test_component/
└── test_component.log
```
## Advanced Usage
### Age-Based Log Cleanup
While the primary cleanup mechanism is count-based (via `max_log_files`), a separate utility function, `cleanup_old_logs`, is available for age-based cleanup if you have specific retention policies.
```python
from utils.logger import cleanup_old_logs
# Deletes all log files in the 'bot_manager' directory older than 15 days
cleanup_old_logs('bot_manager', days_to_keep=15)
```
### Shutting Down Logging
In some cases, especially in tests or when an application is shutting down gracefully, you may need to explicitly close all log file handlers.
```python
from utils.logger import shutdown_logging
# Closes all open file handlers managed by the logging system
shutdown_logging()
```
## Component Integration Pattern (Conditional Logging)
While the logger utility is simple, it is designed to support a powerful conditional logging pattern at the application level. This allows components to be developed to run with or without logging, making them more flexible and easier to test.
### Key Concepts
1. **Optional Logging**: Components are designed to accept `logger=None` in their constructor and function normally without producing any logs.
2. **Error-Only Mode**: A component can be designed to only log messages of level `ERROR` or higher. This is a component-level implementation pattern, not a feature of `get_logger`.
3. **Logger Inheritance**: Parent components can pass their logger instance to child components, ensuring a consistent logging context.
### Example: Component Constructor
All major components should follow this pattern to support conditional logging.
```python
class ComponentExample:
def __init__(self, logger=None, log_errors_only=False):
self.logger = logger
self.log_errors_only = log_errors_only
def _log_info(self, message: str) -> None:
"""Log info message if logger is available and not in errors-only mode."""
if self.logger and not self.log_errors_only:
self.logger.info(message)
def _log_error(self, message: str, exc_info: bool = False) -> None:
"""Log error message if logger is available."""
if self.logger:
self.logger.error(message, exc_info=exc_info)
# ... other helper methods for debug, warning, critical ...
```
This pattern decouples the component's logic from the global logging configuration and makes its logging behavior explicit and easy to manage.
## Troubleshooting
- **Permissions**: Ensure the application has write permissions to the `logs/` directory.
- **No Logs**: If file logging fails (e.g., due to permissions), a warning is printed to the console. If `verbose` is not enabled, no further logs will be produced. Ensure the `logs/` directory is writable.
- **Console Spam**: If the console is too noisy, set `verbose=False` when calling `get_logger` and ensure `VERBOSE_LOGGING` is not set to `true` in your environment.
## Best Practices
### 1. Component Naming