Implement data collection architecture with modular components

- Introduced a comprehensive data collection framework, including `CollectorServiceConfig`, `BaseDataCollector`, and `CollectorManager`, enhancing modularity and maintainability.
- Developed `CollectorFactory` for streamlined collector creation, promoting separation of concerns and improved configuration handling.
- Enhanced `DataCollectionService` to utilize the new architecture, ensuring robust error handling and logging practices.
- Added `TaskManager` for efficient management of asynchronous tasks, improving performance and resource management.
- Implemented health monitoring and auto-recovery features in `CollectorManager`, ensuring reliable operation of data collectors.
- Updated imports across the codebase to reflect the new structure, ensuring consistent access to components.

These changes significantly improve the architecture and maintainability of the data collection service, aligning with project standards for modularity, performance, and error handling.
This commit is contained in:
Vasily.onl
2025-06-10 13:40:28 +08:00
parent c28e4a9aaf
commit f6cb1485b1
18 changed files with 384 additions and 45 deletions

View File

@@ -10,7 +10,8 @@ from typing import Dict, List, Optional, Any, Type, Tuple
from dataclasses import dataclass, field
from utils.logger import get_logger
from ..base_collector import BaseDataCollector, DataType
from ..collector.base_collector import BaseDataCollector
from ..common.data_types import DataType
from ..common import CandleProcessingConfig
from .registry import EXCHANGE_REGISTRY, get_supported_exchanges, get_exchange_info
from .exceptions import (

View File

@@ -11,7 +11,7 @@ from datetime import datetime, timezone
from typing import Dict, List, Optional, Any
from dataclasses import dataclass
from ...base_collector import (
from ...collector.base_collector import (
BaseDataCollector, DataType, CollectorStatus, MarketDataPoint,
OHLCVData, DataValidationError, ConnectionError
)

View File

@@ -11,7 +11,7 @@ from decimal import Decimal
from typing import Dict, List, Optional, Any, Union, Tuple
from enum import Enum
from ...base_collector import DataType, MarketDataPoint
from ...collector.base_collector import DataType, MarketDataPoint
from ...common import (
DataValidationResult,
StandardizedTrade,