- Added `realtime_execution.py` for real-time strategy execution, enabling live signal generation and integration with the dashboard's chart refresh cycle.
- Introduced `data_integration.py` to manage market data orchestration, caching, and technical indicator calculations for strategy signal generation.
- Implemented `validation.py` for comprehensive validation and quality assessment of strategy-generated signals, ensuring reliability and consistency.
- Developed `batch_processing.py` to facilitate efficient backtesting of multiple strategies across large datasets with memory management and performance optimization.
- Updated `__init__.py` files to include new modules and ensure proper exports, enhancing modularity and maintainability.
- Enhanced unit tests for the new features, ensuring robust functionality and adherence to project standards.
These changes establish a solid foundation for real-time strategy execution and data integration, aligning with project goals for modularity, performance, and maintainability.
- Added `StrategyRun` and `StrategySignal` models to track strategy execution sessions and generated signals, respectively, ensuring a clear separation from live trading data.
- Introduced `StrategyRepository` for managing database operations related to strategy runs and signals, including methods for creating, updating, and retrieving strategy data.
- Updated `DatabaseOperations` to integrate the new repository, enhancing the overall architecture and maintaining consistency with existing database access patterns.
- Enhanced documentation to reflect the new database schema and repository functionalities, ensuring clarity for future development and usage.
These changes establish a robust foundation for strategy analysis and backtesting, aligning with project goals for modularity, performance, and maintainability.
- Introduced `config_utils.py` for loading and managing strategy configurations, including functions for loading templates, generating dropdown options, and retrieving parameter schemas and default values.
- Added JSON templates for EMA Crossover, MACD, and RSI strategies, defining their parameters and validation rules to enhance modularity and maintainability.
- Implemented `StrategyManager` in `manager.py` for managing user-defined strategies with file-based storage, supporting easy sharing and portability.
- Updated `__init__.py` to include new components and ensure proper module exports.
- Enhanced error handling and logging practices across the new modules for improved reliability.
These changes establish a robust foundation for strategy management and configuration, aligning with project goals for modularity, performance, and maintainability.
- Introduced a new `strategies` package containing the core structure for trading strategies, including `BaseStrategy`, `StrategyFactory`, and various strategy implementations (EMA, RSI, MACD).
- Added utility functions for signal detection and validation in `strategies/utils.py`, enhancing modularity and maintainability.
- Updated `pyproject.toml` to include the new `strategies` package in the build configuration.
- Implemented comprehensive unit tests for the strategy foundation components, ensuring reliability and adherence to project standards.
These changes establish a solid foundation for the strategy engine, aligning with project goals for modularity, performance, and maintainability.
- Introduced a new `system_health_constants.py` file to define thresholds and constants for system health metrics.
- Refactored existing system health callbacks into modular components, enhancing maintainability and clarity.
- Implemented dynamic loading of time range options in `charts.py`, improving flexibility in time range selection.
- Added detailed documentation for new callback functions, ensuring clarity on their purpose and usage.
- Enhanced error handling and logging practices across the new modules to ensure robust monitoring and debugging capabilities.
These changes significantly improve the architecture and maintainability of the system health monitoring features, aligning with project standards for modularity and performance.
- Introduced dynamic generation of parameter fields and callback handling for indicators, enhancing modularity and maintainability.
- Updated `config_utils.py` with new utility functions to load indicator templates and generate dynamic outputs and states for parameter fields.
- Refactored `indicators.py` to utilize these utilities, streamlining the callback logic and improving user experience by reducing hardcoded elements.
- Modified `indicator_modal.py` to create parameter fields dynamically based on JSON templates, eliminating the need for manual updates when adding new indicators.
- Added documentation outlining the new data-driven architecture for indicators, improving clarity and guidance for future development.
These changes significantly enhance the flexibility and scalability of the indicator system, aligning with project goals for maintainability and performance.
- Introduced `config_utils.py` to provide utility functions for loading and managing indicator templates, enhancing modularity and maintainability.
- Implemented functions to load templates, generate dropdown options, and retrieve parameter schemas, default parameters, and styling for various indicators.
- Updated the indicator modal to dynamically create parameter fields based on the loaded configurations, improving user experience and reducing redundancy.
- Refactored existing parameter field creation logic to utilize the new utility functions, streamlining the codebase and adhering to project standards for clarity and maintainability.
These changes significantly enhance the configuration management for indicators, aligning with project goals for modularity and performance.
- Deleted `app_new.py`, which was previously the main entry point for the dashboard application, to streamline the codebase.
- Consolidated the application initialization and callback registration logic into `main.py`, enhancing modularity and maintainability.
- Updated the logging and error handling practices in `main.py` to ensure consistent application behavior and improved debugging capabilities.
These changes simplify the application structure, aligning with project standards for modularity and maintainability.
- Updated `pyproject.toml` to include the new `data` package in the build configuration, ensuring all components are properly included.
- Introduced `ADR-004` documentation outlining the rationale for refactoring the data collection system into a modular architecture, addressing complexity and maintainability issues.
- Enhanced `data_collectors.md` to reflect the new component structure, detailing responsibilities of `CollectorLifecycleManager`, `ManagerHealthMonitor`, `ManagerStatsTracker`, and `ManagerLogger`.
- Refactored `DataCollectionService` to utilize the new modular components, improving orchestration and error handling.
- Removed the obsolete `collector-service-tasks-optimization.md` and `refactor-common-package.md` files, streamlining the tasks documentation.
These changes significantly improve the architecture and maintainability of the data collection service, aligning with project standards for modularity, performance, and documentation clarity.
- Introduced a comprehensive data collection framework, including `CollectorServiceConfig`, `BaseDataCollector`, and `CollectorManager`, enhancing modularity and maintainability.
- Developed `CollectorFactory` for streamlined collector creation, promoting separation of concerns and improved configuration handling.
- Enhanced `DataCollectionService` to utilize the new architecture, ensuring robust error handling and logging practices.
- Added `TaskManager` for efficient management of asynchronous tasks, improving performance and resource management.
- Implemented health monitoring and auto-recovery features in `CollectorManager`, ensuring reliable operation of data collectors.
- Updated imports across the codebase to reflect the new structure, ensuring consistent access to components.
These changes significantly improve the architecture and maintainability of the data collection service, aligning with project standards for modularity, performance, and error handling.
- Implemented `_sanitize_error` method in `DataCollectionService` and `CollectorManager` to prevent leaking internal error details.
- Improved error handling across various methods by catching specific exceptions and logging sanitized messages with `exc_info=True`.
- Added file permission validation in `ServiceConfig` to ensure secure configuration file handling, including detailed logging for permission issues.
- Refactored logging practices to enhance clarity and maintainability, ensuring consistent error reporting.
These changes significantly bolster the security and robustness of the data collection services, aligning with project standards for error handling and maintainability.
- Introduced `service_config.py` to manage configuration loading, validation, and schema management, enhancing modularity and security.
- Created a `ServiceConfig` class for handling configuration with robust error handling and default values.
- Refactored `DataCollectionService` to utilize the new `ServiceConfig`, streamlining configuration management and improving readability.
- Added a `CollectorFactory` to encapsulate collector creation logic, promoting separation of concerns.
- Updated `CollectorManager` and related components to align with the new architecture, ensuring better maintainability.
- Enhanced logging practices across the service for improved monitoring and debugging.
These changes significantly improve the architecture and maintainability of the data collection service, aligning with project standards for modularity and performance.
- Extracted `OHLCVData` and validation logic into a new `common/ohlcv_data.py` module, promoting better organization and reusability.
- Updated `BaseDataCollector` to utilize the new `validate_ohlcv_data` function for improved data validation, enhancing code clarity and maintainability.
- Refactored imports in `data/__init__.py` to reflect the new structure, ensuring consistent access to common data types and exceptions.
- Removed redundant data validation logic from `BaseDataCollector`, streamlining its responsibilities.
- Added unit tests for `OHLCVData` and validation functions to ensure correctness and reliability.
These changes improve the architecture of the data module, aligning with project standards for maintainability and performance.
- Extracted callback management logic into a new `CallbackDispatcher` class, promoting separation of concerns and enhancing modularity.
- Updated `BaseDataCollector` to utilize the `CallbackDispatcher` for adding, removing, and notifying data callbacks, improving code clarity and maintainability.
- Refactored related methods to ensure consistent error handling and logging practices.
- Added unit tests for the `CallbackDispatcher` to validate its functionality and ensure robust error handling.
These changes streamline the callback management architecture, aligning with project standards for maintainability and performance.
- Extracted connection management logic into a new `ConnectionManager` class, promoting separation of concerns and enhancing modularity.
- Updated `BaseDataCollector` to utilize the `ConnectionManager` for connection, disconnection, and reconnection processes, improving code clarity and maintainability.
- Refactored connection-related methods and attributes, ensuring consistent error handling and logging practices.
- Enhanced the `OKXCollector` to implement the new connection management approach, streamlining its connection logic.
- Added unit tests for the `ConnectionManager` to validate its functionality and ensure robust error handling.
These changes improve the architecture of the data collector, aligning with project standards for maintainability and performance.
- Introduced a new `CollectorStateAndTelemetry` class to encapsulate the status, health checks, and statistics of the data collector, promoting modularity and separation of concerns.
- Updated `BaseDataCollector` to replace direct status management with calls to the new telemetry class, enhancing maintainability and readability.
- Refactored logging methods to utilize the telemetry class, ensuring consistent logging practices.
- Modified the `OKXCollector` to integrate with the new telemetry system for improved status reporting and error handling.
- Added comprehensive tests for the `CollectorStateAndTelemetry` class to ensure functionality and reliability.
These changes streamline the data collector's architecture, aligning with project standards for maintainability and performance.
- Updated all technical indicators to return pandas DataFrames instead of lists, improving consistency and usability.
- Modified the `calculate` method in `TechnicalIndicators` to directly return DataFrames with relevant indicator values.
- Enhanced the `data_integration.py` to utilize the new DataFrame outputs for better integration with charting.
- Updated documentation to reflect the new DataFrame-centric approach, including usage examples and output structures.
- Improved error handling to ensure empty DataFrames are returned when insufficient data is available.
These changes streamline the indicator calculations and improve the overall architecture, aligning with project standards for maintainability and performance.
- Deleted the `okx_config.json` file as part of the configuration refactor.
- Updated `BaseDataCollector` to include an optional `timeframes` parameter for more flexible data collection.
- Modified `DataCollectionService` and `OKXCollector` to pass and utilize the new `timeframes` parameter.
- Enhanced `ExchangeCollectorConfig` to validate timeframes, ensuring they are provided and correctly formatted.
- Updated documentation to reflect the new configurable timeframes feature, improving clarity for users.
These changes streamline the configuration process and improve the flexibility of data collection, aligning with project standards for maintainability and usability.
- Introduced a new `exceptions.py` file containing custom exceptions for the exchanges module, improving error specificity and handling.
- Updated the `factory.py` and `registry.py` files to utilize the new exceptions, enhancing robustness in error reporting and logging.
- Implemented validation logic in `ExchangeCollectorConfig` to ensure proper configuration, raising appropriate exceptions when validation fails.
- Enhanced logging throughout the factory methods to provide better insights into the collector creation process and error scenarios.
- Added comprehensive documentation for the exchanges module, detailing the architecture, error handling, and usage examples.
These changes significantly improve the error handling and maintainability of the exchanges module, aligning with project standards and enhancing developer experience.
- Introduced a new transformation module that includes safety limits for trade operations, enhancing data integrity and preventing errors.
- Refactored existing transformation logic into dedicated classes and functions, improving modularity and maintainability.
- Added detailed validation for trade sizes, prices, and symbol formats, ensuring compliance with trading rules.
- Implemented logging for significant operations and validation checks, aiding in monitoring and debugging.
- Created a changelog to document the new features and changes, providing clarity for future development.
- Developed extensive unit tests to cover the new functionality, ensuring reliability and preventing regressions.
These changes significantly enhance the architecture of the transformation module, making it more robust and easier to manage.
- Removed the existing `validation.py` file and replaced it with a modular structure, introducing separate files for validation results, field validators, and the base validator class.
- Implemented comprehensive validation functions for common data types, enhancing reusability and maintainability.
- Added a new `__init__.py` to expose the validation utilities, ensuring a clean public interface.
- Created detailed documentation for the validation module, including usage examples and architectural details.
- Introduced extensive unit tests to cover the new validation framework, ensuring reliability and preventing regressions.
These changes enhance the overall architecture of the data validation module, making it more scalable and easier to manage.
- Updated project rules to unify structure and interaction with tools, emphasizing the use of UV for package management and Windows PowerShell for terminal commands.
- Added guidelines for argument validation and referencing documentation files for context and architecture.
- Improved links in the README documentation for better navigation and accessibility to project resources.
These changes aim to streamline project management and enhance clarity for developers, ensuring adherence to best practices and coding standards.
- Introduced a dedicated sub-package for technical indicators under `data/common/indicators/`, improving modularity and maintainability.
- Moved `TechnicalIndicators` and `IndicatorResult` classes to their respective files, along with utility functions for configuration management.
- Updated import paths throughout the codebase to reflect the new structure, ensuring compatibility.
- Added comprehensive safety net tests for the indicators module to verify core functionality and prevent regressions during refactoring.
- Enhanced documentation to provide clear usage examples and details on the new package structure.
These changes improve the overall architecture of the technical indicators module, making it more scalable and easier to manage.
- Split the `aggregation.py` file into a dedicated sub-package, improving modularity and maintainability.
- Moved `TimeframeBucket`, `RealTimeCandleProcessor`, and `BatchCandleProcessor` classes into their respective files within the new `aggregation` sub-package.
- Introduced utility functions for trade aggregation and validation, enhancing code organization.
- Updated import paths throughout the codebase to reflect the new structure, ensuring compatibility.
- Added safety net tests for the aggregation package to verify core functionality and prevent regressions during refactoring.
These changes enhance the overall architecture of the aggregation module, making it more scalable and easier to manage.
- Replaced the `RedisManager` class with a more modular `SyncRedisManager` and `AsyncRedisManager`, improving the separation of synchronous and asynchronous operations.
- Updated the `system_health.py` callbacks to utilize the new `get_sync_redis_manager` function for Redis interactions, simplifying the connection process.
- Enhanced error handling and logging in Redis status checks, providing clearer feedback on connection issues.
- Revised the setup documentation to reflect changes in Redis connection testing, ensuring clarity for users.
These updates improve the maintainability and reliability of Redis interactions within the system, aligning with best practices for modular design.
- Removed the `RawDataManager` class and integrated its functionality directly into the `RawTradeRepository`, streamlining the management of raw trade data.
- Implemented the `cleanup_old_raw_data` method to delete outdated records, preventing table bloat and improving performance.
- Added the `get_raw_data_stats` method to retrieve statistics about raw data storage, enhancing data management capabilities.
- Updated documentation to reflect the new methods and their usage, ensuring clarity for future developers.
These changes improve the maintainability and efficiency of the database operations related to raw trade data.
- Updated the `MarketDataRepository` and `RawTradeRepository` classes to exclusively utilize SQLAlchemy ORM for all database interactions, enhancing maintainability and type safety.
- Removed raw SQL queries in favor of ORM methods, ensuring a consistent and database-agnostic approach across the repository layer.
- Revised documentation to reflect these changes, emphasizing the importance of using the ORM for database operations.
These modifications improve the overall architecture of the database layer, making it more scalable and easier to manage.
- Introduced a modular repository structure by creating separate repository classes for `Bot`, `MarketData`, and `RawTrade`, improving code organization and maintainability.
- Updated the `DatabaseOperations` class to utilize the new repository classes, enhancing the abstraction of database interactions.
- Refactored the `.env` file to update database connection parameters and add new logging and health monitoring configurations.
- Modified the `okx_config.json` to change default timeframes for trading pairs, aligning with updated requirements.
- Added comprehensive unit tests for the new repository classes, ensuring robust functionality and reliability.
These changes improve the overall architecture of the database layer, making it more scalable and easier to manage.
- Updated the logging documentation to reflect changes in the unified log message format, including the addition of pathname, line number, and function name for better traceability.
- Modified the `get_logger` function to set a default value for `component_name`, improving usability for users who may not specify a component name.
- Ensured consistency in the documentation regarding the parameters and their descriptions.
These updates improve the clarity and ease of use of the logging system, making it more accessible for developers.
- Revised the logging documentation to clarify the unified logging system's features and usage patterns.
- Simplified the logger implementation by removing the custom `DateRotatingFileHandler` and utilizing the standard library's `TimedRotatingFileHandler` for date-based log rotation.
- Enhanced the `get_logger` function to ensure thread-safe logger configuration and prevent duplicate handlers.
- Introduced a new `cleanup_old_logs` function for age-based log cleanup, while retaining the existing count-based cleanup mechanism.
- Improved error handling and logging setup to ensure robust logging behavior across components.
These changes enhance the clarity and maintainability of the logging system, making it easier for developers to implement and utilize logging in their components.
