vasily/TCPDashboard
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor data module to enhance modularity and maintainability

Browse Source 
- Extracted `OHLCVData` and validation logic into a new `common/ohlcv_data.py` module, promoting better organization and reusability.
- Updated `BaseDataCollector` to utilize the new `validate_ohlcv_data` function for improved data validation, enhancing code clarity and maintainability.
- Refactored imports in `data/__init__.py` to reflect the new structure, ensuring consistent access to common data types and exceptions.
- Removed redundant data validation logic from `BaseDataCollector`, streamlining its responsibilities.
- Added unit tests for `OHLCVData` and validation functions to ensure correctness and reliability.

These changes improve the architecture of the data module, aligning with project standards for maintainability and performance.
This commit is contained in:
Vasily.onl
2025-06-10 12:04:58 +08:00
parent 3db8fb1c41
commit 33f2110f19
15 changed files with 511 additions and 1009 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/modules/exchanges/exchanges.md
View File

@@ -6,8 +6,8 @@
1. **Base Collector**
- Inherit from `BaseDataCollector`
- Implement required abstract methods
- Handle connection lifecycle
- Implement exchange-specific abstract methods (e.g., `_actual_connect`, `_actual_disconnect`, `_subscribe_channels`, `_process_message`)
- Leverage `ConnectionManager`, `CollectorStateAndTelemetry`, and `CallbackDispatcher` through the inherited `BaseDataCollector` functionalities
2. **WebSocket Client**
- Implement exchange-specific WebSocket handling

36
docs/modules/exchanges/okx_collector.md
View File

@@ -897,13 +897,13 @@ The OKX collector consists of three main components working together:
### `OKXCollector`
- **Main class**: `OKXCollector(BaseDataCollector)`
- **Responsibilities**:
- Manages WebSocket connection state
- Subscribes to required data channels
- Dispatches raw messages to the data processor
- Stores standardized data in the database
- Provides health and status monitoring
- **Main class**: `OKXCollector(BaseDataCollector)`
- **Responsibilities**:
- Implements exchange-specific connection and subscription logic (delegating to `ConnectionManager` for core connection handling).
- Processes and standardizes raw OKX WebSocket messages (delegating to `OKXDataProcessor`).
- Interacts with `CollectorStateAndTelemetry` for status, health, and logging.
- Uses `CallbackDispatcher` to notify subscribers of processed data.
- Stores standardized data in the database.
### `OKXWebSocketClient`
@@ -915,12 +915,12 @@ The OKX collector consists of three main components working together:
### `OKXDataProcessor`
- **New in v2.0**: `OKXDataProcessor`
- **Responsibilities**:
- Validates incoming raw data from WebSocket
- Transforms data into standardized `StandardizedTrade` and `OHLCVCandle` formats
- Aggregates trades into OHLCV candles
- Invokes callbacks for processed trades and completed candles
- **New in v2.0**: `OKXDataProcessor`
- **Responsibilities**:
- Validates incoming raw data from WebSocket.
- Transforms data into standardized `MarketDataPoint` and `OHLCVData` formats (using the moved `OHLCVData`).
- Aggregates trades into OHLCV candles.
- Invokes callbacks for processed trades and completed candles.
## Configuration
@@ -932,12 +932,12 @@ Configuration options for the `OKXCollector` class:
|-------------------------|---------------------|---------------------------------------|-----------------------------------------------------------------------------|
| `symbol` | `str` | - | Trading symbol (e.g., `BTC-USDT`) |
| `data_types` | `List[DataType]` | `[TRADE, ORDERBOOK]` | List of data types to collect |
| `auto_restart` | `bool` | `True` | Automatically restart on failures |
| `health_check_interval` | `float` | `30.0` | Seconds between health checks |
| `auto_restart` | `bool` | `True` | Automatically restart on failures (managed by `BaseDataCollector` via `ConnectionManager`) |
| `health_check_interval` | `float` | `30.0` | Seconds between health checks (managed by `BaseDataCollector` via `CollectorStateAndTelemetry`) |
| `store_raw_data` | `bool` | `True` | Store raw WebSocket data for debugging |
| `force_update_candles` | `bool` | `False` | If `True`, update existing candles; if `False`, keep existing ones unchanged |
| `logger` | `Logger` | `None` | Logger instance for conditional logging |
| `log_errors_only` | `bool` | `False` | If `True` and logger provided, only log error-level messages |
| `logger` | `Logger` | `None` | Logger instance for conditional logging (managed by `BaseDataCollector` via `CollectorStateAndTelemetry`) |
| `log_errors_only` | `bool` | `False` | If `True` and logger provided, only log error-level messages (managed by `BaseDataCollector` via `CollectorStateAndTelemetry`) |
### Health & Status Monitoring
@@ -962,4 +962,4 @@ Example output:
}
```
## Database Integration
## Database Integration