vasily/TCPDashboard
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor data collection architecture for modularity and maintainability

Browse Source 
- Updated `pyproject.toml` to include the new `data` package in the build configuration, ensuring all components are properly included.
- Introduced `ADR-004` documentation outlining the rationale for refactoring the data collection system into a modular architecture, addressing complexity and maintainability issues.
- Enhanced `data_collectors.md` to reflect the new component structure, detailing responsibilities of `CollectorLifecycleManager`, `ManagerHealthMonitor`, `ManagerStatsTracker`, and `ManagerLogger`.
- Refactored `DataCollectionService` to utilize the new modular components, improving orchestration and error handling.
- Removed the obsolete `collector-service-tasks-optimization.md` and `refactor-common-package.md` files, streamlining the tasks documentation.

These changes significantly improve the architecture and maintainability of the data collection service, aligning with project standards for modularity, performance, and documentation clarity.
This commit is contained in:
Vasily.onl
2025-06-10 14:32:00 +08:00
parent f6cb1485b1
commit 0a7e444206
7 changed files with 505 additions and 444 deletions
Download Patch File Download Diff File Expand all files Collapse all files

131
tasks/collector-service-tasks-optimization.md
View File

@@ -1,131 +0,0 @@
## Relevant Files
- `data/collector_manager.py` - Core manager for data collectors (refactored: 563→178 lines, enhanced with TaskManager).
- `data/collection_service.py` - Main service for data collection (enhanced with TaskManager).
- `data/collector_types.py` - Shared data types for collector management (new file).
- `data/manager_components/` - Component classes for modular manager architecture (new directory).
- `data/manager_components/manager_stats_tracker.py` - Enhanced with performance monitoring and cache optimization.
- `utils/async_task_manager.py` - New comprehensive async task management utility (new file).
- `data/__init__.py` - Updated imports for new structure.
- `tests/test_collector_manager.py` - Unit tests for `collector_manager.py` (imports updated).
- `tests/test_data_collection_aggregation.py` - Integration tests (imports updated).
- `scripts/production_clean.py` - Production script (verified working).
- `scripts/start_data_collection.py` - Data collection script (verified working).
## Code Review Analysis: `collection_service.py` & `collector_manager.py`
### Overall Assessment
Both files show good foundational architecture but exceed the recommended file size limits and contain several areas for improvement.
### 📏 File Size Violations
- **`collector_manager.py`**: 563 lines (❌ Exceeds 250-line limit by 125%)
- **`collection_service.py`**: 451 lines (❌ Exceeds 250-line limit by 80%)
### 🔍 Function Size Analysis
**Functions Exceeding 50-Line Limit:**
**`collector_manager.py`:**
- `__init__()` - 65 lines
- `_global_health_monitor()` - 71 lines
- `get_status()` - 53 lines
**`collection_service.py`:**
- `_create_default_config()` - 89 lines
- `run()` - 98 lines
### 🏗️ Architecture & Design Issues
1. **Tight Coupling in CollectorManager**
- **Issue**: The manager class handles too many responsibilities (collector lifecycle, health monitoring, statistics, logging).
- **Solution**: Apply Single Responsibility Principle by creating dedicated component classes.
2. **Configuration Management Complexity**
- **Issue**: Configuration logic scattered across multiple methods.
- **Solution**: Dedicated configuration manager for centralized handling.
### 🔒 Security & Error Handling Review
**Strengths:**
- Proper exception handling with context
- No hardcoded credentials
- Graceful shutdown handling
- Input validation in configuration
**Areas for Improvement:**
1. **Error Information Leakage**
- **Issue**: Could leak internal details.
- **Solution**: Sanitize error messages before logging.
2. **Configuration File Security**
- **Issue**: No file permission validation.
- **Solution**: Add validation to ensure appropriate file permissions.
### 🚀 Performance Optimization Opportunities
1. **Async Task Management**
- **Issue**: Potential memory leaks with untracked tasks.
- **Solution**: Implement proper task lifecycle management with a `TaskManager`.
2. **Statistics Collection Optimization**
- **Issue**: Statistics calculated on every status request.
- **Solution**: Use cached statistics with background updates via a `CachedStatsManager`.
### 🧪 Testing & Maintainability
**Missing Test Coverage Areas:**
1. Collector manager state transitions
2. Health monitoring edge cases
3. Configuration validation
4. Signal handling
5. Concurrent collector operations
### 📝 Documentation Improvements
1. **Missing API Documentation**
- **Issue**: Public methods and classes lack comprehensive docstrings.
- **Solution**: Add examples, thread safety, and performance considerations.
2. **Configuration Schema Documentation**
- **Issue**: No formal schema validation.
- **Solution**: Implement JSON schema validation for configurations.
### 📊 Quality Metrics Summary
| Metric | Current | Target | Status |
|--------|---------|--------|--------|
| File Size | 563/451 lines | <250 lines | ❌ |
| Function Size | 5 functions >50 lines | 0 functions >50 lines | ❌ |
| Cyclomatic Complexity | Medium-High | Low-Medium | ⚠️ |
| Test Coverage | ~30% estimated | >80% | ❌ |
| Documentation | Basic | Comprehensive | ⚠️ |
| Error Handling | Good | Excellent | ✅ |
## Tasks
- [x] 1.0 Refactor `collector_manager.py` for Modularity and Readability
- [x] 1.1 Extract `ManagerStatus` and `CollectorConfig` dataclasses to `data/collector_types.py`.
- [x] 1.2 Create `data/manager_components/collector_lifecycle_manager.py` to handle `add_collector`, `remove_collector`, `enable_collector`, `disable_collector`, `_start_collector`, `restart_collector`, `restart_all_collectors`.
- [x] 1.3 Create `data/manager_components/manager_health_monitor.py` to encapsulate `_global_health_monitor` logic.
- [x] 1.4 Create `data/manager_components/manager_stats_tracker.py` to manage statistics in `get_status` and update `_stats`.
- [x] 1.5 Create `data/manager_components/manager_logger.py` to centralize logging methods (`_log_debug`, `_log_info`, `_log_warning`, `_log_error`, `_log_critical`).
- [x] 1.6 Update `CollectorManager` to use instances of these new component classes.
- [x] 1.7 Ensure `CollectorManager` `__init__` method is under 50 lines by delegating initialization to helper methods within the class or component classes.
- [x] 2.0 Refactor `collection_service.py` for Improved Structure
- [x] 2.1 Create `config/service_config.py` to handle `_load_config` and `_create_default_config` logic, including schema validation.
- [x] 2.2 Create `data/collector_factory.py` to encapsulate `_create_collector` logic.
- [x] 2.3 Update `DataCollectionService` to use instances of these new component classes.
- [x] 2.4 Refactor `run()` method to be under 50 lines by extracting sub-logics (e.g., `_run_main_loop`).
- [x] 2.5 Test './scripts/start_data_collection.py' and './scripts/production_clean.py' to ensure they work as expected.
- [x] 3.0 Enhance Error Handling and Security
- [x] 3.1 Implement a `_sanitize_error` method in `CollectorManager` and `DataCollectionService` to prevent leaking internal error details.
- [x] 3.2 Add file permission validation for configuration files in `config/service_config.py`.
- [x] 3.3 Review all `try-except` blocks to ensure specific exceptions are caught rather than broad `Exception`.
- [x] 3.4 Ensure all logger calls include `exc_info=True` for error and critical logs.
- [x] 3.5 Test './scripts/start_data_collection.py' and './scripts/production_clean.py' to ensure they work as expected.
- [x] 4.0 Optimize Performance and Resource Management
- [x] 4.1 Implement a `TaskManager` class in `utils/async_task_manager.py` to manage and track `asyncio.Task` instances in `CollectorManager` and `DataCollectionService`.
- [x] 4.2 Introduce a `CachedStatsManager` in `data/manager_components/manager_stats_tracker.py` for `CollectorManager` to cache statistics and update them periodically instead of on every `get_status` call.
- [x] 4.3 Review all `asyncio.sleep` calls for optimal intervals.
- [x] 4.4 Test './scripts/start_data_collection.py' and './scripts/production_clean.py' to ensure they work as expected.
- [ ] 5.0 Improve Documentation and Test Coverage
- [ ] 5.1 Add comprehensive docstrings to all public methods and classes in `CollectorManager` and `DataCollectionService`, including examples, thread safety notes, and performance considerations.
- [ ] 5.2 Create new unit test files: `tests/data/manager_components/test_collector_lifecycle_manager.py`, `tests/data/manager_components/test_manager_health_monitor.py`, `tests/data/manager_components/test_manager_stats_tracker.py`, `tests/config/test_service_config.py`, `tests/data/test_collector_factory.py`.
- [ ] 5.3 Write unit tests for all new components (lifecycle manager, health monitor, stats tracker, service config, collector factory).
- [ ] 5.4 Enhance existing tests or create new ones for `CollectorManager` to cover state transitions, health monitoring edge cases, and concurrent operations.
- [ ] 5.5 Enhance existing tests or create new ones for `DataCollectionService` to cover configuration validation, service lifecycle, and signal handling.
- [ ] 5.6 Ensure all tests use `uv run pytest` and verify passing.
- [ ] 5.7 Test './scripts/start_data_collection.py' and './scripts/production_clean.py' to ensure they work as expected.

66
tasks/refactor-common-package.md
View File

@@ -1,66 +0,0 @@
## Relevant Files
- `data/common/aggregation.py` - To be broken into a sub-package.
- `data/common/indicators.py` - To be broken into a sub-package and have a bug fixed.
- `data/common/validation.py` - To be refactored for better modularity.
- `data/common/transformation.py` - ✅ Refactored into transformation package with safety limits.
- `data/common/data_types.py` - To be updated with new types from other modules.
- `data/common/__init__.py` - ✅ Updated to reflect the new package structure.
- `tests/` - Existing tests will need to be run after each step to ensure no regressions.
### Notes
- This refactoring focuses on improving modularity by splitting large files into smaller, more focused modules, as outlined in the `refactoring.mdc` guide.
- Each major step will be followed by a verification phase to ensure the application remains stable.
## Tasks
- [x] 1.0 Refactor `aggregation.py` into a dedicated sub-package.
- [x] 1.1 Create safety net tests to ensure the aggregation logic still works as expected.
- [x] 1.2 Create a new directory `data/common/aggregation`.
- [x] 1.3 Create `data/common/aggregation/__init__.py` to mark it as a package.
- [x] 1.4 Move the `TimeframeBucket` class to `data/common/aggregation/bucket.py`.
- [x] 1.5 Move the `RealTimeCandleProcessor` class to `data/common/aggregation/realtime.py`.
- [x] 1.6 Move the `BatchCandleProcessor` class to `data/common/aggregation/batch.py`.
- [x] 1.7 Move the utility functions to `data/common/aggregation/utils.py`.
- [x] 1.8 Update `data/common/aggregation/__init__.py` to expose all public classes and functions.
- [x] 1.9 Delete the original `data/common/aggregation.py` file.
- [x] 1.10 Run tests to verify the aggregation logic still works as expected.
- [x] 2.0 Refactor `indicators.py` into a dedicated sub-package.
- [x] 2.1 Create safety net tests for indicators module.
- [x] 2.2 Create a new directory `data/common/indicators`.
- [x] 2.3 Create `data/common/indicators/__init__.py` to mark it as a package.
- [x] 2.4 Move the `TechnicalIndicators` class to `data/common/indicators/technical.py`.
- [x] 2.5 Move the `IndicatorResult` class to `data/common/indicators/result.py`.
- [x] 2.6 Move the utility functions to `data/common/indicators/utils.py`.
- [x] 2.7 Update `data/common/indicators/__init__.py` to expose all public classes and functions.
- [x] 2.8 Delete the original `data/common/indicators.py` file.
- [x] 2.9 Run tests to verify the indicators logic still works as expected.
- [x] 3.0 Refactor `validation.py` for better modularity.
- [x] 3.1 Create safety net tests for validation module.
- [x] 3.2 Extract common validation logic into separate functions.
- [x] 3.3 Improve error handling and validation messages.
- [x] 3.4 Run tests to verify validation still works as expected.
- [x] 4.0 Refactor `transformation.py` for better modularity.
- [x] 4.1 Create safety net tests for transformation module.
- [x] 4.2 Extract common transformation logic into separate functions.
- [x] 4.3 Improve error handling and transformation messages.
- [x] 4.4 Run tests to verify transformation still works as expected.
- [x] 4.5 Create comprehensive safety limits system.
- [x] 4.6 Add documentation for the transformation module.
- [x] 4.7 Delete redundant transformation.py file.
- [x] 5.0 Update `data_types.py` with new types.
- [x] 5.1 Review and document all data types.
- [x] 5.2 Add any missing type hints.
- [x] 5.3 Add validation for data types.
- [x] 5.4 Run tests to verify data types still work as expected.
- [ ] 6.0 Final verification and cleanup.
- [x] 6.1 Run all tests to ensure no regressions.
- [x] 6.2 Update documentation to reflect new structure.
- [x] 6.3 Review and clean up any remaining TODOs.
- [ ] 6.4 Create PR with changes.

66
tasks/tasks-base-collector-refactoring.md
View File

@@ -1,66 +0,0 @@
## Relevant Files
- `data/base_collector.py` - The main file to be refactored, where `BaseDataCollector` is defined.
- `data/collector/collector_state_telemetry.py` - New file for managing collector status, health, and statistics.
- `data/collector/collector_connection_manager.py` - New file for handling connection, disconnection, and reconnection logic.
- `data/collector/collector_callback_dispatcher.py` - New file for managing data callbacks and notifications.
- `data/ohlcv_data.py` - Potential new file for `OHLCVData` and related validation if deemed beneficial.
- `tests/data/test_base_collector.py` - Existing test file for `BaseDataCollector`.
- `tests/data/collector/test_collector_state_telemetry.py` - New test file for `CollectorStateAndTelemetry` class.
- `tests/data/collector/test_collector_connection_manager.py` - New test file for `ConnectionManager` class.
- `tests/data/collector/test_collector_callback_dispatcher.py` - New test file for `CallbackDispatcher` class.
- `tests/data/test_ohlcv_data.py` - New test file for `OHLCVData` and validation.
### Notes
- Unit tests should typically be placed alongside the code files they are testing (e.g., `MyComponent.tsx` and `MyComponent.test.tsx` in the same directory).
- Each refactoring step will be small and verified with existing tests, and new tests will be created for extracted components.
## Tasks
- [x] 0.0 Create `data/collector` directory
- [x] 1.0 Extract `CollectorStateAndTelemetry` Class
- [x] 1.1 Create `data/collector/collector_state_telemetry.py`.
- [x] 1.2 Move `CollectorStatus` enum to `data/collector/collector_state_telemetry.py`.
- [x] 1.3 Move `_stats` initialization and related helper methods (`_log_debug`, `_log_info`, `_log_warning`, `_log_error`, `_log_critical`) to `CollectorStateAndTelemetry`.
- [x] 1.4 Move `get_status` and `get_health_status` methods to `CollectorStateAndTelemetry`.
- [x] 1.5 Implement a constructor for `CollectorStateAndTelemetry` to receive logger and initial parameters.
- [x] 1.6 Add necessary imports to both `data/base_collector.py` and `data/collector/collector_state_telemetry.py`.
- [x] 1.7 Create `tests/data/collector/test_collector_state_telemetry.py` and add initial tests for the new class.
- [x] 2.0 Extract `ConnectionManager` Class
- [x] 2.1 Create `data/collector/collector_connection_manager.py`.
- [x] 2.2 Move connection-related attributes (`_connection`, `_reconnect_attempts`, `_max_reconnect_attempts`, `_reconnect_delay`) to `ConnectionManager`.
- [x] 2.3 Move `connect`, `disconnect`, `_handle_connection_error` methods to `ConnectionManager`.
- [x] 2.4 Implement a constructor for `ConnectionManager` to receive logger and other necessary parameters.
- [x] 2.5 Add necessary imports to both `data/base_collector.py` and `data/collector/collector_connection_manager.py`.
- [x] 2.6 Create `tests/data/collector/test_collector_connection_manager.py` and add initial tests for the new class.
- [x] 3.0 Extract `CallbackDispatcher` Class
- [x] 3.1 Create `data/collector/collector_callback_dispatcher.py`.
- [x] 3.2 Move `_data_callbacks` attribute to `CallbackDispatcher`.
- [x] 3.3 Move `add_data_callback`, `remove_data_callback`, `_notify_callbacks` methods to `CallbackDispatcher`.
- [x] 3.4 Implement a constructor for `CallbackDispatcher` to receive logger.
- [x] 3.5 Add necessary imports to both `data/base_collector.py` and `data/collector/collector_callback_dispatcher.py`.
- [x] 3.6 Create `tests/data/collector/test_collector_callback_dispatcher.py` and add initial tests for the new class.
- [x] 4.0 Refactor `BaseDataCollector` to use new components
- [x] 4.1 Update `BaseDataCollector.__init__` to instantiate and use `CollectorStateAndTelemetry`, `ConnectionManager`, and `CallbackDispatcher` instances.
- [x] 4.2 Replace direct access to moved attributes/methods with calls to the new component instances (e.g., `self.logger.info` becomes `self._state_telemetry.log_info`).
- [x] 4.3 Modify `start`, `stop`, `restart`, `_message_loop`, `_health_monitor` to interact with the new components, delegating responsibilities appropriately.
- [x] 4.4 Update `get_status` and `get_health_status` in `BaseDataCollector` to delegate to `CollectorStateAndTelemetry`.
- [x] 4.5 Review and update abstract methods and their calls as needed, ensuring they interact correctly with the new components.
- [x] 4.6 Ensure all existing tests for `BaseDataCollector` still pass after refactoring.
- [x] 4.7 Update `data/exchanges/okx/collector.py` to use the new `CollectorStateAndTelemetry` and `ConnectionManager` classes for logging, status updates, and connection handling.
- [x] 4.8 Update `data/collector_manager.py` to interact with the new `CollectorStateAndTelemetry` class for health checks and status retrieval from `BaseDataCollector` instances.
- [x] 5.0 Review and potentially extract `OHLCVData` and related validation
- [x] 5.1 Analyze if `OHLCVData` and `validate_ohlcv_data` are frequently used outside of `data/base_collector.py`.
- [x] 5.2 If analysis indicates external usage or clear separation benefits, move `OHLCVData` class and `DataValidationError` to a new `data/ohlcv_data.py` file.
- [x] 5.3 Update imports in `data/base_collector.py` and any other affected files.
- [x] 5.4 If `OHLCVData` is extracted, create `tests/data/test_ohlcv_data.py` with tests for its structure and validation logic.
- [x] 6.0 Update Module Imports
- [x] 6.1 Update imports in `data/__init__.py` to reflect the new locations of `CollectorStatus`, `DataCollectorError`, `DataValidationError`, `DataType`, `MarketDataPoint`, and `OHLCVData` (if moved).
- [x] 6.2 Update imports in `data/common/data_types.py` for `DataType` and `MarketDataPoint`.
- [x] 6.3 Review and update imports in all test files (`tests/test_refactored_okx.py`, `tests/test_real_storage.py`, `tests/test_okx_collector.py`, `tests/test_exchange_factory.py`, `tests/test_data_collection_aggregation.py`, `tests/test_collector_manager.py`, `tests/test_base_collector.py`, `tests/database/test_database_operations.py`) and scripts (`scripts/production_clean.py`) that import directly from `data.base_collector`.