# ADR-004: Modular Data Collector System Refactoring

## Status
Accepted

## Context
Previously, the data collection system, primarily `CollectorManager` and `DataCollectionService`, had grown in complexity, exceeding recommended file and function size limits. This led to tight coupling, scattered configuration logic, and a monolithic structure that hindered maintainability, testability, and scalability. Key issues included:

- Large file sizes for `collector_manager.py` (563 lines) and `collection_service.py` (451 lines).
- Functions exceeding the 50-line limit (e.g., `__init__`, `_global_health_monitor`, `get_status` in `CollectorManager`; `_create_default_config`, `run` in `DataCollectionService`).
- `CollectorManager` handling too many responsibilities (lifecycle, health monitoring, statistics, logging).
- Scattered configuration logic.
- Potential memory leaks due to untracked asynchronous tasks.
- Inefficient statistics collection.
- Gaps in test coverage for state transitions, health monitoring, and concurrent operations.
- Lack of comprehensive API and configuration schema documentation.

## Decision
We decided to refactor the data collector system into a modular, component-based architecture. This involves:

1.  **Breaking Down `CollectorManager`**: Extracting specific responsibilities into dedicated component classes:
    *   `CollectorLifecycleManager`: For collector lifecycle operations.
    *   `ManagerHealthMonitor`: For global health monitoring.
    *   `ManagerStatsTracker`: For managing and caching performance statistics.
    *   `ManagerLogger`: For centralizing logging.
2.  **Modularizing `DataCollectionService`**: Creating specialized components for its concerns:
    *   `ServiceConfig`: For loading, creating, and validating configurations.
    *   `CollectorFactory`: For encapsulating collector creation logic.
3.  **Introducing `AsyncTaskManager`**: A centralized utility for managing and tracking `asyncio.Task` instances to prevent resource leaks and improve robustness.
4.  **Enhancing Error Handling and Security**: Implementing a `_sanitize_error` method, adding file permission validation for configurations, and ensuring specific exception handling.
5.  **Optimizing Performance**: Utilizing `CachedStatsManager` for efficient statistics updates.
6.  **Improving Documentation and Testing**: Adding comprehensive docstrings, creating new unit test files for components, and enhancing existing tests.

## Consequences

**Positive:**
-   **Improved Maintainability**: Clear separation of concerns reduces complexity and makes code easier to understand and modify.
-   **Enhanced Testability**: Individual components can be unit-tested in isolation, leading to more robust and reliable code.
-   **Increased Scalability**: The modular design allows for easier extension and adaptation to new exchanges or data types.
-   **Better Readability**: Smaller files and functions improve code comprehension.
-   **Robust Error Handling**: Dedicated components for error handling and task management lead to more resilient operations.
-   **Optimized Performance**: Cached statistics and managed asynchronous tasks contribute to better resource utilization.
-   **Comprehensive Documentation**: Clearer architecture facilitates better documentation and onboarding.

**Negative:**
-   **Increased File Count**: More files are introduced due to the breakdown of responsibilities.
-   **Initial Development Overhead**: The refactoring required a significant upfront investment in time and effort.
-   **Increased Indirection**: More layers of abstraction might initially make the system seem more complex to new developers unfamiliar with the pattern.

## Alternatives Considered
-   **Minor Refactoring**: Only addressing critical violations (e.g., function size) without a full modular redesign. *Rejected* due to not fully addressing underlying architectural issues and long-term maintainability concerns.
-   **External Libraries for Each Concern**: Using separate, heavy-duty external libraries for logging, health monitoring, etc. *Rejected* to avoid introducing unnecessary dependencies and to maintain more control over custom logic specific to our system.