## Relevant Files - `data/base_collector.py` - The main file to be refactored, where `BaseDataCollector` is defined. - `data/collector/collector_state_telemetry.py` - New file for managing collector status, health, and statistics. - `data/collector/collector_connection_manager.py` - New file for handling connection, disconnection, and reconnection logic. - `data/collector/collector_callback_dispatcher.py` - New file for managing data callbacks and notifications. - `data/ohlcv_data.py` - Potential new file for `OHLCVData` and related validation if deemed beneficial. - `tests/data/test_base_collector.py` - Existing test file for `BaseDataCollector`. - `tests/data/collector/test_collector_state_telemetry.py` - New test file for `CollectorStateAndTelemetry` class. - `tests/data/collector/test_collector_connection_manager.py` - New test file for `ConnectionManager` class. - `tests/data/collector/test_collector_callback_dispatcher.py` - New test file for `CallbackDispatcher` class. - `tests/data/test_ohlcv_data.py` - New test file for `OHLCVData` and validation. ### Notes - Unit tests should typically be placed alongside the code files they are testing (e.g., `MyComponent.tsx` and `MyComponent.test.tsx` in the same directory). - Each refactoring step will be small and verified with existing tests, and new tests will be created for extracted components. ## Tasks - [x] 0.0 Create `data/collector` directory - [x] 1.0 Extract `CollectorStateAndTelemetry` Class - [x] 1.1 Create `data/collector/collector_state_telemetry.py`. - [x] 1.2 Move `CollectorStatus` enum to `data/collector/collector_state_telemetry.py`. - [x] 1.3 Move `_stats` initialization and related helper methods (`_log_debug`, `_log_info`, `_log_warning`, `_log_error`, `_log_critical`) to `CollectorStateAndTelemetry`. - [x] 1.4 Move `get_status` and `get_health_status` methods to `CollectorStateAndTelemetry`. - [x] 1.5 Implement a constructor for `CollectorStateAndTelemetry` to receive logger and initial parameters. - [x] 1.6 Add necessary imports to both `data/base_collector.py` and `data/collector/collector_state_telemetry.py`. - [x] 1.7 Create `tests/data/collector/test_collector_state_telemetry.py` and add initial tests for the new class. - [x] 2.0 Extract `ConnectionManager` Class - [x] 2.1 Create `data/collector/collector_connection_manager.py`. - [x] 2.2 Move connection-related attributes (`_connection`, `_reconnect_attempts`, `_max_reconnect_attempts`, `_reconnect_delay`) to `ConnectionManager`. - [x] 2.3 Move `connect`, `disconnect`, `_handle_connection_error` methods to `ConnectionManager`. - [x] 2.4 Implement a constructor for `ConnectionManager` to receive logger and other necessary parameters. - [x] 2.5 Add necessary imports to both `data/base_collector.py` and `data/collector/collector_connection_manager.py`. - [x] 2.6 Create `tests/data/collector/test_collector_connection_manager.py` and add initial tests for the new class. - [x] 3.0 Extract `CallbackDispatcher` Class - [x] 3.1 Create `data/collector/collector_callback_dispatcher.py`. - [x] 3.2 Move `_data_callbacks` attribute to `CallbackDispatcher`. - [x] 3.3 Move `add_data_callback`, `remove_data_callback`, `_notify_callbacks` methods to `CallbackDispatcher`. - [x] 3.4 Implement a constructor for `CallbackDispatcher` to receive logger. - [x] 3.5 Add necessary imports to both `data/base_collector.py` and `data/collector/collector_callback_dispatcher.py`. - [x] 3.6 Create `tests/data/collector/test_collector_callback_dispatcher.py` and add initial tests for the new class. - [x] 4.0 Refactor `BaseDataCollector` to use new components - [x] 4.1 Update `BaseDataCollector.__init__` to instantiate and use `CollectorStateAndTelemetry`, `ConnectionManager`, and `CallbackDispatcher` instances. - [x] 4.2 Replace direct access to moved attributes/methods with calls to the new component instances (e.g., `self.logger.info` becomes `self._state_telemetry.log_info`). - [x] 4.3 Modify `start`, `stop`, `restart`, `_message_loop`, `_health_monitor` to interact with the new components, delegating responsibilities appropriately. - [x] 4.4 Update `get_status` and `get_health_status` in `BaseDataCollector` to delegate to `CollectorStateAndTelemetry`. - [x] 4.5 Review and update abstract methods and their calls as needed, ensuring they interact correctly with the new components. - [x] 4.6 Ensure all existing tests for `BaseDataCollector` still pass after refactoring. - [x] 4.7 Update `data/exchanges/okx/collector.py` to use the new `CollectorStateAndTelemetry` and `ConnectionManager` classes for logging, status updates, and connection handling. - [x] 4.8 Update `data/collector_manager.py` to interact with the new `CollectorStateAndTelemetry` class for health checks and status retrieval from `BaseDataCollector` instances. - [x] 5.0 Review and potentially extract `OHLCVData` and related validation - [x] 5.1 Analyze if `OHLCVData` and `validate_ohlcv_data` are frequently used outside of `data/base_collector.py`. - [x] 5.2 If analysis indicates external usage or clear separation benefits, move `OHLCVData` class and `DataValidationError` to a new `data/ohlcv_data.py` file. - [x] 5.3 Update imports in `data/base_collector.py` and any other affected files. - [x] 5.4 If `OHLCVData` is extracted, create `tests/data/test_ohlcv_data.py` with tests for its structure and validation logic. - [x] 6.0 Update Module Imports - [x] 6.1 Update imports in `data/__init__.py` to reflect the new locations of `CollectorStatus`, `DataCollectorError`, `DataValidationError`, `DataType`, `MarketDataPoint`, and `OHLCVData` (if moved). - [x] 6.2 Update imports in `data/common/data_types.py` for `DataType` and `MarketDataPoint`. - [x] 6.3 Review and update imports in all test files (`tests/test_refactored_okx.py`, `tests/test_real_storage.py`, `tests/test_okx_collector.py`, `tests/test_exchange_factory.py`, `tests/test_data_collection_aggregation.py`, `tests/test_collector_manager.py`, `tests/test_base_collector.py`, `tests/database/test_database_operations.py`) and scripts (`scripts/production_clean.py`) that import directly from `data.base_collector`.