vasily/TCPDashboard
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
TCPDashboard/tasks/tasks-base-collector-refactoring.md
Vasily.onl 33f2110f19 Refactor data module to enhance modularity and maintainability 
- Extracted `OHLCVData` and validation logic into a new `common/ohlcv_data.py` module, promoting better organization and reusability.
- Updated `BaseDataCollector` to utilize the new `validate_ohlcv_data` function for improved data validation, enhancing code clarity and maintainability.
- Refactored imports in `data/__init__.py` to reflect the new structure, ensuring consistent access to common data types and exceptions.
- Removed redundant data validation logic from `BaseDataCollector`, streamlining its responsibilities.
- Added unit tests for `OHLCVData` and validation functions to ensure correctness and reliability.

These changes improve the architecture of the data module, aligning with project standards for maintainability and performance.
2025-06-10 12:04:58 +08:00

6.1 KiB
Raw Blame History

Relevant Files

  • data/base_collector.py - The main file to be refactored, where BaseDataCollector is defined.
  • data/collector/collector_state_telemetry.py - New file for managing collector status, health, and statistics.
  • data/collector/collector_connection_manager.py - New file for handling connection, disconnection, and reconnection logic.
  • data/collector/collector_callback_dispatcher.py - New file for managing data callbacks and notifications.
  • data/ohlcv_data.py - Potential new file for OHLCVData and related validation if deemed beneficial.
  • tests/data/test_base_collector.py - Existing test file for BaseDataCollector.
  • tests/data/collector/test_collector_state_telemetry.py - New test file for CollectorStateAndTelemetry class.
  • tests/data/collector/test_collector_connection_manager.py - New test file for ConnectionManager class.
  • tests/data/collector/test_collector_callback_dispatcher.py - New test file for CallbackDispatcher class.
  • tests/data/test_ohlcv_data.py - New test file for OHLCVData and validation.

Notes

  • Unit tests should typically be placed alongside the code files they are testing (e.g., MyComponent.tsx and MyComponent.test.tsx in the same directory).
  • Each refactoring step will be small and verified with existing tests, and new tests will be created for extracted components.

Tasks

  • 0.0 Create data/collector directory

  • 1.0 Extract CollectorStateAndTelemetry Class

    • 1.1 Create data/collector/collector_state_telemetry.py.
    • 1.2 Move CollectorStatus enum to data/collector/collector_state_telemetry.py.
    • 1.3 Move _stats initialization and related helper methods (_log_debug, _log_info, _log_warning, _log_error, _log_critical) to CollectorStateAndTelemetry.
    • 1.4 Move get_status and get_health_status methods to CollectorStateAndTelemetry.
    • 1.5 Implement a constructor for CollectorStateAndTelemetry to receive logger and initial parameters.
    • 1.6 Add necessary imports to both data/base_collector.py and data/collector/collector_state_telemetry.py.
    • 1.7 Create tests/data/collector/test_collector_state_telemetry.py and add initial tests for the new class.

  • 2.0 Extract ConnectionManager Class

    • 2.1 Create data/collector/collector_connection_manager.py.
    • 2.2 Move connection-related attributes (_connection, _reconnect_attempts, _max_reconnect_attempts, _reconnect_delay) to ConnectionManager.
    • 2.3 Move connect, disconnect, _handle_connection_error methods to ConnectionManager.
    • 2.4 Implement a constructor for ConnectionManager to receive logger and other necessary parameters.
    • 2.5 Add necessary imports to both data/base_collector.py and data/collector/collector_connection_manager.py.
    • 2.6 Create tests/data/collector/test_collector_connection_manager.py and add initial tests for the new class.

  • 3.0 Extract CallbackDispatcher Class

    • 3.1 Create data/collector/collector_callback_dispatcher.py.
    • 3.2 Move _data_callbacks attribute to CallbackDispatcher.
    • 3.3 Move add_data_callback, remove_data_callback, _notify_callbacks methods to CallbackDispatcher.
    • 3.4 Implement a constructor for CallbackDispatcher to receive logger.
    • 3.5 Add necessary imports to both data/base_collector.py and data/collector/collector_callback_dispatcher.py.
    • 3.6 Create tests/data/collector/test_collector_callback_dispatcher.py and add initial tests for the new class.

  • 4.0 Refactor BaseDataCollector to use new components

    • 4.1 Update BaseDataCollector.__init__ to instantiate and use CollectorStateAndTelemetry, ConnectionManager, and CallbackDispatcher instances.
    • 4.2 Replace direct access to moved attributes/methods with calls to the new component instances (e.g., self.logger.info becomes self._state_telemetry.log_info).
    • 4.3 Modify start, stop, restart, _message_loop, _health_monitor to interact with the new components, delegating responsibilities appropriately.
    • 4.4 Update get_status and get_health_status in BaseDataCollector to delegate to CollectorStateAndTelemetry.
    • 4.5 Review and update abstract methods and their calls as needed, ensuring they interact correctly with the new components.
    • 4.6 Ensure all existing tests for BaseDataCollector still pass after refactoring.
    • 4.7 Update data/exchanges/okx/collector.py to use the new CollectorStateAndTelemetry and ConnectionManager classes for logging, status updates, and connection handling.
    • 4.8 Update data/collector_manager.py to interact with the new CollectorStateAndTelemetry class for health checks and status retrieval from BaseDataCollector instances.

  • 5.0 Review and potentially extract OHLCVData and related validation

    • 5.1 Analyze if OHLCVData and validate_ohlcv_data are frequently used outside of data/base_collector.py.
    • 5.2 If analysis indicates external usage or clear separation benefits, move OHLCVData class and DataValidationError to a new data/ohlcv_data.py file.
    • 5.3 Update imports in data/base_collector.py and any other affected files.
    • 5.4 If OHLCVData is extracted, create tests/data/test_ohlcv_data.py with tests for its structure and validation logic.

  • 6.0 Update Module Imports

    • 6.1 Update imports in data/__init__.py to reflect the new locations of CollectorStatus, DataCollectorError, DataValidationError, DataType, MarketDataPoint, and OHLCVData (if moved).
    • 6.2 Update imports in data/common/data_types.py for DataType and MarketDataPoint.
    • 6.3 Review and update imports in all test files (tests/test_refactored_okx.py, tests/test_real_storage.py, tests/test_okx_collector.py, tests/test_exchange_factory.py, tests/test_data_collection_aggregation.py, tests/test_collector_manager.py, tests/test_base_collector.py, tests/database/test_database_operations.py) and scripts (scripts/production_clean.py) that import directly from data.base_collector.