3db8fb1c41
- Extracted callback management logic into a new `CallbackDispatcher` class, promoting separation of concerns and enhancing modularity. - Updated `BaseDataCollector` to utilize the `CallbackDispatcher` for adding, removing, and notifying data callbacks, improving code clarity and maintainability. - Refactored related methods to ensure consistent error handling and logging practices. - Added unit tests for the `CallbackDispatcher` to validate its functionality and ensure robust error handling. These changes streamline the callback management architecture, aligning with project standards for maintainability and performance.
66 lines
6.1 KiB
Markdown
66 lines
6.1 KiB
Markdown
## Relevant Files
|
|
|
|
- `data/base_collector.py` - The main file to be refactored, where `BaseDataCollector` is defined.
|
|
- `data/collector/collector_state_telemetry.py` - New file for managing collector status, health, and statistics.
|
|
- `data/collector/collector_connection_manager.py` - New file for handling connection, disconnection, and reconnection logic.
|
|
- `data/collector/collector_callback_dispatcher.py` - New file for managing data callbacks and notifications.
|
|
- `data/ohlcv_data.py` - Potential new file for `OHLCVData` and related validation if deemed beneficial.
|
|
- `tests/data/test_base_collector.py` - Existing test file for `BaseDataCollector`.
|
|
- `tests/data/collector/test_collector_state_telemetry.py` - New test file for `CollectorStateAndTelemetry` class.
|
|
- `tests/data/collector/test_collector_connection_manager.py` - New test file for `ConnectionManager` class.
|
|
- `tests/data/collector/test_collector_callback_dispatcher.py` - New test file for `CallbackDispatcher` class.
|
|
- `tests/data/test_ohlcv_data.py` - New test file for `OHLCVData` and validation.
|
|
|
|
### Notes
|
|
|
|
- Unit tests should typically be placed alongside the code files they are testing (e.g., `MyComponent.tsx` and `MyComponent.test.tsx` in the same directory).
|
|
- Each refactoring step will be small and verified with existing tests, and new tests will be created for extracted components.
|
|
|
|
## Tasks
|
|
|
|
- [ ] 0.0 Create `data/collector` directory
|
|
- [x] 1.0 Extract `CollectorStateAndTelemetry` Class
|
|
- [x] 1.1 Create `data/collector/collector_state_telemetry.py`.
|
|
- [x] 1.2 Move `CollectorStatus` enum to `data/collector/collector_state_telemetry.py`.
|
|
- [x] 1.3 Move `_stats` initialization and related helper methods (`_log_debug`, `_log_info`, `_log_warning`, `_log_error`, `_log_critical`) to `CollectorStateAndTelemetry`.
|
|
- [x] 1.4 Move `get_status` and `get_health_status` methods to `CollectorStateAndTelemetry`.
|
|
- [x] 1.5 Implement a constructor for `CollectorStateAndTelemetry` to receive logger and initial parameters.
|
|
- [x] 1.6 Add necessary imports to both `data/base_collector.py` and `data/collector/collector_state_telemetry.py`.
|
|
- [x] 1.7 Create `tests/data/collector/test_collector_state_telemetry.py` and add initial tests for the new class.
|
|
|
|
- [x] 2.0 Extract `ConnectionManager` Class
|
|
- [x] 2.1 Create `data/collector/collector_connection_manager.py`.
|
|
- [x] 2.2 Move connection-related attributes (`_connection`, `_reconnect_attempts`, `_max_reconnect_attempts`, `_reconnect_delay`) to `ConnectionManager`.
|
|
- [x] 2.3 Move `connect`, `disconnect`, `_handle_connection_error` methods to `ConnectionManager`.
|
|
- [x] 2.4 Implement a constructor for `ConnectionManager` to receive logger and other necessary parameters.
|
|
- [x] 2.5 Add necessary imports to both `data/base_collector.py` and `data/collector/collector_connection_manager.py`.
|
|
- [x] 2.6 Create `tests/data/collector/test_collector_connection_manager.py` and add initial tests for the new class.
|
|
|
|
- [x] 3.0 Extract `CallbackDispatcher` Class
|
|
- [x] 3.1 Create `data/collector/collector_callback_dispatcher.py`.
|
|
- [x] 3.2 Move `_data_callbacks` attribute to `CallbackDispatcher`.
|
|
- [x] 3.3 Move `add_data_callback`, `remove_data_callback`, `_notify_callbacks` methods to `CallbackDispatcher`.
|
|
- [x] 3.4 Implement a constructor for `CallbackDispatcher` to receive logger.
|
|
- [x] 3.5 Add necessary imports to both `data/base_collector.py` and `data/collector/collector_callback_dispatcher.py`.
|
|
- [x] 3.6 Create `tests/data/collector/test_collector_callback_dispatcher.py` and add initial tests for the new class.
|
|
|
|
- [ ] 4.0 Refactor `BaseDataCollector` to use new components
|
|
- [ ] 4.1 Update `BaseDataCollector.__init__` to instantiate and use `CollectorStateAndTelemetry`, `ConnectionManager`, and `CallbackDispatcher` instances.
|
|
- [ ] 4.2 Replace direct access to moved attributes/methods with calls to the new component instances (e.g., `self.logger.info` becomes `self._state_telemetry.log_info`).
|
|
- [ ] 4.3 Modify `start`, `stop`, `restart`, `_message_loop`, `_health_monitor` to interact with the new components, delegating responsibilities appropriately.
|
|
- [ ] 4.4 Update `get_status` and `get_health_status` in `BaseDataCollector` to delegate to `CollectorStateAndTelemetry`.
|
|
- [ ] 4.5 Review and update abstract methods and their calls as needed, ensuring they interact correctly with the new components.
|
|
- [ ] 4.6 Ensure all existing tests for `BaseDataCollector` still pass after refactoring.
|
|
- [ ] 4.7 Update `data/exchanges/okx/collector.py` to use the new `CollectorStateAndTelemetry` and `ConnectionManager` classes for logging, status updates, and connection handling.
|
|
- [ ] 4.8 Update `data/collector_manager.py` to interact with the new `CollectorStateAndTelemetry` class for health checks and status retrieval from `BaseDataCollector` instances.
|
|
|
|
- [ ] 5.0 Review and potentially extract `OHLCVData` and related validation
|
|
- [ ] 5.1 Analyze if `OHLCVData` and `validate_ohlcv_data` are frequently used outside of `data/base_collector.py`.
|
|
- [ ] 5.2 If analysis indicates external usage or clear separation benefits, move `OHLCVData` class and `DataValidationError` to a new `data/ohlcv_data.py` file.
|
|
- [ ] 5.3 Update imports in `data/base_collector.py` and any other affected files.
|
|
- [ ] 5.4 If `OHLCVData` is extracted, create `tests/data/test_ohlcv_data.py` with tests for its structure and validation logic.
|
|
|
|
- [ ] 6.0 Update Module Imports
|
|
- [ ] 6.1 Update imports in `data/__init__.py` to reflect the new locations of `CollectorStatus`, `DataCollectorError`, `DataValidationError`, `DataType`, `MarketDataPoint`, and `OHLCVData` (if moved).
|
|
- [ ] 6.2 Update imports in `data/common/data_types.py` for `DataType` and `MarketDataPoint`.
|
|
- [ ] 6.3 Review and update imports in all test files (`tests/test_refactored_okx.py`, `tests/test_real_storage.py`, `tests/test_okx_collector.py`, `tests/test_exchange_factory.py`, `tests/test_data_collection_aggregation.py`, `tests/test_collector_manager.py`, `tests/test_base_collector.py`, `tests/database/test_database_operations.py`) and scripts (`scripts/production_clean.py`) that import directly from `data.base_collector`. |