- Added `realtime_execution.py` for real-time strategy execution, enabling live signal generation and integration with the dashboard's chart refresh cycle.
- Introduced `data_integration.py` to manage market data orchestration, caching, and technical indicator calculations for strategy signal generation.
- Implemented `validation.py` for comprehensive validation and quality assessment of strategy-generated signals, ensuring reliability and consistency.
- Developed `batch_processing.py` to facilitate efficient backtesting of multiple strategies across large datasets with memory management and performance optimization.
- Updated `__init__.py` files to include new modules and ensure proper exports, enhancing modularity and maintainability.
- Enhanced unit tests for the new features, ensuring robust functionality and adherence to project standards.
These changes establish a solid foundation for real-time strategy execution and data integration, aligning with project goals for modularity, performance, and maintainability.
- Added `StrategyRun` and `StrategySignal` models to track strategy execution sessions and generated signals, respectively, ensuring a clear separation from live trading data.
- Introduced `StrategyRepository` for managing database operations related to strategy runs and signals, including methods for creating, updating, and retrieving strategy data.
- Updated `DatabaseOperations` to integrate the new repository, enhancing the overall architecture and maintaining consistency with existing database access patterns.
- Enhanced documentation to reflect the new database schema and repository functionalities, ensuring clarity for future development and usage.
These changes establish a robust foundation for strategy analysis and backtesting, aligning with project goals for modularity, performance, and maintainability.
- Introduced `config_utils.py` for loading and managing strategy configurations, including functions for loading templates, generating dropdown options, and retrieving parameter schemas and default values.
- Added JSON templates for EMA Crossover, MACD, and RSI strategies, defining their parameters and validation rules to enhance modularity and maintainability.
- Implemented `StrategyManager` in `manager.py` for managing user-defined strategies with file-based storage, supporting easy sharing and portability.
- Updated `__init__.py` to include new components and ensure proper module exports.
- Enhanced error handling and logging practices across the new modules for improved reliability.
These changes establish a robust foundation for strategy management and configuration, aligning with project goals for modularity, performance, and maintainability.
- Introduced a new `strategies` package containing the core structure for trading strategies, including `BaseStrategy`, `StrategyFactory`, and various strategy implementations (EMA, RSI, MACD).
- Added utility functions for signal detection and validation in `strategies/utils.py`, enhancing modularity and maintainability.
- Updated `pyproject.toml` to include the new `strategies` package in the build configuration.
- Implemented comprehensive unit tests for the strategy foundation components, ensuring reliability and adherence to project standards.
These changes establish a solid foundation for the strategy engine, aligning with project goals for modularity, performance, and maintainability.
- Updated `pyproject.toml` to include the new `data` package in the build configuration, ensuring all components are properly included.
- Introduced `ADR-004` documentation outlining the rationale for refactoring the data collection system into a modular architecture, addressing complexity and maintainability issues.
- Enhanced `data_collectors.md` to reflect the new component structure, detailing responsibilities of `CollectorLifecycleManager`, `ManagerHealthMonitor`, `ManagerStatsTracker`, and `ManagerLogger`.
- Refactored `DataCollectionService` to utilize the new modular components, improving orchestration and error handling.
- Removed the obsolete `collector-service-tasks-optimization.md` and `refactor-common-package.md` files, streamlining the tasks documentation.
These changes significantly improve the architecture and maintainability of the data collection service, aligning with project standards for modularity, performance, and documentation clarity.
- Introduced a comprehensive data collection framework, including `CollectorServiceConfig`, `BaseDataCollector`, and `CollectorManager`, enhancing modularity and maintainability.
- Developed `CollectorFactory` for streamlined collector creation, promoting separation of concerns and improved configuration handling.
- Enhanced `DataCollectionService` to utilize the new architecture, ensuring robust error handling and logging practices.
- Added `TaskManager` for efficient management of asynchronous tasks, improving performance and resource management.
- Implemented health monitoring and auto-recovery features in `CollectorManager`, ensuring reliable operation of data collectors.
- Updated imports across the codebase to reflect the new structure, ensuring consistent access to components.
These changes significantly improve the architecture and maintainability of the data collection service, aligning with project standards for modularity, performance, and error handling.
- Implemented `_sanitize_error` method in `DataCollectionService` and `CollectorManager` to prevent leaking internal error details.
- Improved error handling across various methods by catching specific exceptions and logging sanitized messages with `exc_info=True`.
- Added file permission validation in `ServiceConfig` to ensure secure configuration file handling, including detailed logging for permission issues.
- Refactored logging practices to enhance clarity and maintainability, ensuring consistent error reporting.
These changes significantly bolster the security and robustness of the data collection services, aligning with project standards for error handling and maintainability.
- Introduced `service_config.py` to manage configuration loading, validation, and schema management, enhancing modularity and security.
- Created a `ServiceConfig` class for handling configuration with robust error handling and default values.
- Refactored `DataCollectionService` to utilize the new `ServiceConfig`, streamlining configuration management and improving readability.
- Added a `CollectorFactory` to encapsulate collector creation logic, promoting separation of concerns.
- Updated `CollectorManager` and related components to align with the new architecture, ensuring better maintainability.
- Enhanced logging practices across the service for improved monitoring and debugging.
These changes significantly improve the architecture and maintainability of the data collection service, aligning with project standards for modularity and performance.
- Extracted `OHLCVData` and validation logic into a new `common/ohlcv_data.py` module, promoting better organization and reusability.
- Updated `BaseDataCollector` to utilize the new `validate_ohlcv_data` function for improved data validation, enhancing code clarity and maintainability.
- Refactored imports in `data/__init__.py` to reflect the new structure, ensuring consistent access to common data types and exceptions.
- Removed redundant data validation logic from `BaseDataCollector`, streamlining its responsibilities.
- Added unit tests for `OHLCVData` and validation functions to ensure correctness and reliability.
These changes improve the architecture of the data module, aligning with project standards for maintainability and performance.
- Extracted callback management logic into a new `CallbackDispatcher` class, promoting separation of concerns and enhancing modularity.
- Updated `BaseDataCollector` to utilize the `CallbackDispatcher` for adding, removing, and notifying data callbacks, improving code clarity and maintainability.
- Refactored related methods to ensure consistent error handling and logging practices.
- Added unit tests for the `CallbackDispatcher` to validate its functionality and ensure robust error handling.
These changes streamline the callback management architecture, aligning with project standards for maintainability and performance.
- Extracted connection management logic into a new `ConnectionManager` class, promoting separation of concerns and enhancing modularity.
- Updated `BaseDataCollector` to utilize the `ConnectionManager` for connection, disconnection, and reconnection processes, improving code clarity and maintainability.
- Refactored connection-related methods and attributes, ensuring consistent error handling and logging practices.
- Enhanced the `OKXCollector` to implement the new connection management approach, streamlining its connection logic.
- Added unit tests for the `ConnectionManager` to validate its functionality and ensure robust error handling.
These changes improve the architecture of the data collector, aligning with project standards for maintainability and performance.
- Introduced a new `CollectorStateAndTelemetry` class to encapsulate the status, health checks, and statistics of the data collector, promoting modularity and separation of concerns.
- Updated `BaseDataCollector` to replace direct status management with calls to the new telemetry class, enhancing maintainability and readability.
- Refactored logging methods to utilize the telemetry class, ensuring consistent logging practices.
- Modified the `OKXCollector` to integrate with the new telemetry system for improved status reporting and error handling.
- Added comprehensive tests for the `CollectorStateAndTelemetry` class to ensure functionality and reliability.
These changes streamline the data collector's architecture, aligning with project standards for maintainability and performance.
- Updated all technical indicators to return pandas DataFrames instead of lists, improving consistency and usability.
- Modified the `calculate` method in `TechnicalIndicators` to directly return DataFrames with relevant indicator values.
- Enhanced the `data_integration.py` to utilize the new DataFrame outputs for better integration with charting.
- Updated documentation to reflect the new DataFrame-centric approach, including usage examples and output structures.
- Improved error handling to ensure empty DataFrames are returned when insufficient data is available.
These changes streamline the indicator calculations and improve the overall architecture, aligning with project standards for maintainability and performance.
- Introduced a new transformation module that includes safety limits for trade operations, enhancing data integrity and preventing errors.
- Refactored existing transformation logic into dedicated classes and functions, improving modularity and maintainability.
- Added detailed validation for trade sizes, prices, and symbol formats, ensuring compliance with trading rules.
- Implemented logging for significant operations and validation checks, aiding in monitoring and debugging.
- Created a changelog to document the new features and changes, providing clarity for future development.
- Developed extensive unit tests to cover the new functionality, ensuring reliability and preventing regressions.
These changes significantly enhance the architecture of the transformation module, making it more robust and easier to manage.
- Removed the existing `validation.py` file and replaced it with a modular structure, introducing separate files for validation results, field validators, and the base validator class.
- Implemented comprehensive validation functions for common data types, enhancing reusability and maintainability.
- Added a new `__init__.py` to expose the validation utilities, ensuring a clean public interface.
- Created detailed documentation for the validation module, including usage examples and architectural details.
- Introduced extensive unit tests to cover the new validation framework, ensuring reliability and preventing regressions.
These changes enhance the overall architecture of the data validation module, making it more scalable and easier to manage.
- Introduced a dedicated sub-package for technical indicators under `data/common/indicators/`, improving modularity and maintainability.
- Moved `TechnicalIndicators` and `IndicatorResult` classes to their respective files, along with utility functions for configuration management.
- Updated import paths throughout the codebase to reflect the new structure, ensuring compatibility.
- Added comprehensive safety net tests for the indicators module to verify core functionality and prevent regressions during refactoring.
- Enhanced documentation to provide clear usage examples and details on the new package structure.
These changes improve the overall architecture of the technical indicators module, making it more scalable and easier to manage.
- Split the `aggregation.py` file into a dedicated sub-package, improving modularity and maintainability.
- Moved `TimeframeBucket`, `RealTimeCandleProcessor`, and `BatchCandleProcessor` classes into their respective files within the new `aggregation` sub-package.
- Introduced utility functions for trade aggregation and validation, enhancing code organization.
- Updated import paths throughout the codebase to reflect the new structure, ensuring compatibility.
- Added safety net tests for the aggregation package to verify core functionality and prevent regressions during refactoring.
These changes enhance the overall architecture of the aggregation module, making it more scalable and easier to manage.
- Enhanced the `UserIndicator` class to include an optional `timeframe` attribute for custom indicator timeframes.
- Updated the `get_indicator_data` method in `MarketDataIntegrator` to fetch and calculate indicators based on the specified timeframe, ensuring proper data alignment and handling.
- Modified the `ChartBuilder` to pass the correct DataFrame for plotting indicators with different timeframes.
- Added UI elements in the indicator modal for selecting timeframes, improving user experience.
- Updated relevant JSON templates to include the new `timeframe` field for all indicators.
- Refactored the `prepare_chart_data` function to ensure it returns a DataFrame with a `DatetimeIndex` for consistent calculations.
This commit enhances the flexibility and usability of the indicator system, allowing users to analyze data across various timeframes.
- Added type conversion for relevant columns in `VolumeAnalyzer` and `PriceMovementAnalyzer` to ensure consistent data handling and avoid type errors.
- Refactored the `create_data_analysis_panel` function to implement a tabbed interface for volume and price movement analysis, improving user experience and organization of analysis tools.
- Updated styles in `indicator_modal.py` for better layout and responsiveness of the modal component.
- Marked unit testing of dashboard components as complete in the task list.
- Updated `ChartBuilder` to support dynamic indicator integration, allowing users to specify overlay and subplot indicators for enhanced chart analysis.
- Implemented a new `get_indicator_data` method in `MarketDataIntegrator` for fetching indicator data based on user configurations.
- Added `create_export_controls` in `chart_controls.py` to facilitate data export options (CSV/JSON) for user analysis.
- Enhanced error handling and logging throughout the chart and data analysis processes to improve reliability and user feedback.
- Updated documentation to reflect new features and usage guidelines for indicator management and data export functionalities.
- Updated `app_new.py` to run the application in debug mode for stability.
- Introduced a new time range control panel in `dashboard/components/chart_controls.py`, allowing users to select predefined time ranges and custom date ranges.
- Enhanced chart callbacks in `dashboard/callbacks/charts.py` to handle time range inputs, ensuring accurate market statistics and analysis based on user selections.
- Implemented logic to preserve chart state during updates, preventing resets of zoom/pan settings.
- Updated market statistics display to reflect the selected time range, improving user experience and data relevance.
- Added a clear button for custom date ranges to reset selections easily.
- Enhanced documentation to reflect the new time range features and usage guidelines.
- Added `psutil` dependency for system performance metrics.
- Implemented a new layout in `dashboard/layouts/system_health.py` using Mantine components for real-time monitoring of data collection services, database health, Redis status, and system performance.
- Enhanced callbacks in `dashboard/callbacks/system_health.py` for detailed status updates and error handling.
- Introduced quick status indicators for data collection, database, Redis, and performance metrics with auto-refresh functionality.
- Created modals for viewing detailed data collection information and service logs.
- Updated documentation to reflect the new features and usage guidelines.
- Introduced `BotIntegratedSignalLayer` and `BotIntegratedTradeLayer` to facilitate automated data fetching and visualization of bot signals and trades.
- Implemented `BotDataService` for efficient retrieval of bot-related data, including filtering and performance summaries.
- Added support for various bot-enhanced layers, including support/resistance and custom strategy layers, to improve trading analysis.
- Updated existing signal layer components to integrate with the new bot functionalities, ensuring seamless operation.
- Enhanced logging and error handling for better debugging and user feedback during bot operations.
- Included comprehensive tests for new functionalities to ensure reliability and maintainability.
- Updated documentation to reflect the new bot integration features and usage guidelines.
- Introduced `TradingSignalLayer` and `TradeExecutionLayer` for visualizing buy/sell signals and trade entries/exits on charts.
- Implemented signal validation and filtering mechanisms to ensure data integrity and user-configurable options.
- Enhanced market data layout to support new timeframes for improved user experience.
- Updated documentation to reflect the new signal layer architecture and its integration with the dashboard.
- Ensured compatibility with existing components while maintaining a modular structure for future enhancements.
- Deleted `app.py`, consolidating the main application logic into a modular structure for improved maintainability.
- Added `dash-mantine-components` dependency to enhance UI component capabilities.
- Updated `pyproject.toml` and `uv.lock` to reflect the new dependency.
- Adjusted imports in `components/__init__.py` and `chart_controls.py` to align with the new modular design.
- Cleaned up unused parameter controls in the market data layout to streamline the user interface.
- Introduced a new modular structure for the dashboard, enhancing maintainability and scalability.
- Created main application entry point in `app_new.py`, integrating all components and callbacks.
- Developed layout modules for market data, bot management, performance analytics, and system health in the `layouts` directory.
- Implemented callback modules for navigation, charts, indicators, and system health in the `callbacks` directory.
- Established reusable UI components in the `components` directory, including chart controls and indicator modals.
- Enhanced documentation to reflect the new modular structure and provide clear usage guidelines.
- Ensured all components are under 300-400 lines for better readability and maintainability.
- Introduced a comprehensive user indicator management system in `components/charts/indicator_manager.py`, allowing users to create, edit, and manage custom indicators with JSON persistence.
- Added new default indicators in `components/charts/indicator_defaults.py` to provide users with immediate options for technical analysis.
- Enhanced the chart rendering capabilities by implementing the `create_chart_with_indicators` function in `components/charts/builder.py`, supporting both overlay and subplot indicators.
- Updated the main application layout in `app.py` to include a modal for adding and editing indicators, improving user interaction.
- Enhanced documentation to cover the new indicator system, including a quick guide for adding new indicators and detailed usage examples.
- Added unit tests to ensure the reliability and functionality of the new indicator management features.
Implement comprehensive chart configuration and validation system
- Introduced a modular chart configuration system in `components/charts/config/` to manage indicator definitions, default configurations, and strategy-specific setups.
- Added new modules for error handling and validation, enhancing user guidance and error reporting capabilities.
- Implemented detailed schema validation for indicators and strategies, ensuring robust configuration management.
- Created example strategies and default configurations to facilitate user onboarding and usage.
- Enhanced documentation to provide clear guidelines on the configuration system, validation rules, and usage examples.
- Added unit tests for all new components to ensure functionality and reliability across the configuration system.
Implement modular chart layers and error handling for Crypto Trading Bot Dashboard
- Introduced a comprehensive chart layer system in `components/charts/layers/` to support various technical indicators and subplots.
- Added base layer components including `BaseLayer`, `CandlestickLayer`, and `VolumeLayer` for flexible chart rendering.
- Implemented overlay indicators such as `SMALayer`, `EMALayer`, and `BollingerBandsLayer` with robust error handling.
- Created subplot layers for indicators like `RSILayer` and `MACDLayer`, enhancing visualization capabilities.
- Developed a `MarketDataIntegrator` for seamless data fetching and validation, improving data quality assurance.
- Enhanced error handling utilities in `components/charts/error_handling.py` to manage insufficient data scenarios effectively.
- Updated documentation to reflect the new chart layer architecture and usage guidelines.
- Added unit tests for all chart layer components to ensure functionality and reliability.
- Suppressed SQLAlchemy logging in `app.py` and `main.py` to reduce console verbosity.
- Introduced a new modular chart system in `components/charts/` with a `ChartBuilder` class for flexible chart creation.
- Added utility functions for data processing and validation in `components/charts/utils.py`.
- Implemented indicator definitions and configurations in `components/charts/config/indicator_defs.py`.
- Created a comprehensive documentation structure for the new chart system, ensuring clarity and maintainability.
- Added unit tests for the `ChartBuilder` class to verify functionality and robustness.
- Updated existing components to integrate with the new chart system, enhancing overall architecture and user experience.
- Introduced `app.py` as the main entry point for the dashboard, providing real-time visualization and bot management interface.
- Implemented layout components including header, navigation tabs, and content areas for market data, bot management, performance analytics, and system health.
- Added callbacks for dynamic updates of market data charts and statistics, ensuring real-time interaction.
- Created reusable UI components in `components` directory for modularity and maintainability.
- Enhanced database operations for fetching market data and checking data availability.
- Updated `main.py` to start the dashboard application with improved user instructions and error handling.
- Documented components and functions for clarity and future reference.
- Marked task 2.9 as complete in the project documentation by adding comprehensive unit tests for data collection and aggregation functionality.
- Created `test_data_collection_aggregation.py` to cover OKX data collection, real-time candle aggregation, data validation, and transformation.
- Included tests for error handling, edge cases, and performance to ensure robustness and reliability of the data processing components.
- Enhanced documentation within the test module to provide clarity on the testing approach and coverage.
- Deleted `example_complete_series_aggregation.py` as it is no longer needed.
- Introduced `data_collection_service.py`, a production-ready service for cryptocurrency market data collection with clean logging and robust error handling.
- Added configuration management for multiple trading pairs and exchanges, supporting health monitoring and graceful shutdown.
- Created `data_collection.json` for service configuration, including exchange settings and logging preferences.
- Updated `CandleProcessingConfig` to reflect changes in timeframes for candle processing.
- Enhanced documentation to cover the new data collection service and its configuration, ensuring clarity for users.
- Introduced `indicators.py` containing implementations for SMA, EMA, RSI, MACD, and Bollinger Bands, optimized for handling sparse OHLCV data.
- Added `IndicatorResult` dataclass to encapsulate results of indicator calculations.
- Implemented methods for calculating multiple indicators efficiently with JSON configuration support and validation.
- Updated `__init__.py` to include new indicators in the module's exports.
- Enhanced documentation to cover the new technical indicators module, including usage examples and integration details.
- Added comprehensive unit tests to ensure accuracy and robustness of the indicators module.
- Added logger parameter to the OKXCollector to enable detailed ping/pong logging.
- Updated message processing methods to maintain heartbeat and track data reception timestamps.
- Adjusted ProductionManager to disable auto-restart and enable full logging for debugging WebSocket issues.
- Enhanced overall logging capabilities to facilitate better monitoring and troubleshooting of data collection processes.
- Introduced `monitor_clean.py` for monitoring database status with detailed logging and status updates.
- Added `production_clean.py` for running OKX data collection with minimal console output and comprehensive logging.
- Implemented command-line argument parsing for both scripts to customize monitoring intervals and collection durations.
- Enhanced logging capabilities to provide clear insights into data collection and monitoring processes.
- Updated documentation to include usage examples and descriptions for the new scripts, ensuring clarity for users.
- Introduced a modular architecture for data processing, including common utilities for validation, transformation, and aggregation.
- Implemented `StandardizedTrade`, `OHLCVCandle`, and `TimeframeBucket` classes for unified data handling across exchanges.
- Developed `OKXDataProcessor` for OKX-specific data validation and processing, leveraging the new common framework.
- Enhanced `OKXCollector` to utilize the common data processing utilities, improving modularity and maintainability.
- Updated documentation to reflect the new architecture and provide guidance on the data processing framework.
- Created comprehensive tests for the new data processing components to ensure reliability and functionality.
- Introduced the `OKXCollector` and `OKXWebSocketClient` classes for real-time market data collection from the OKX exchange.
- Implemented a factory pattern for creating exchange-specific collectors, enhancing modularity and scalability.
- Added configuration support for the OKX collector in `config/okx_config.json`.
- Updated documentation to reflect the new modular architecture and provide guidance on using the OKX collector.
- Created unit tests for the OKX collector and exchange factory to ensure functionality and reliability.
- Enhanced logging and error handling throughout the new implementation for improved monitoring and debugging.
- Introduced `BaseDataCollector` and `CollectorManager` classes for standardized data collection and centralized management.
- Added health monitoring features, including auto-restart capabilities and detailed status reporting for collectors.
- Updated `env.template` to include new logging and health check configurations.
- Enhanced documentation in `docs/data_collectors.md` to provide comprehensive guidance on the new data collection system.
- Added unit tests for `BaseDataCollector` and `CollectorManager` to ensure reliability and functionality.
- Updated `.gitignore` to exclude log files from version control.
- Added `pytest` as a dependency in `pyproject.toml` for testing purposes.
- Included `pytest` in `uv.lock` to ensure consistent dependency management.
- Introduced comprehensive documentation for the new unified logging system in `docs/logging.md`, detailing features, usage, and configuration options.
- Introduced `alembic.ini` for Alembic configuration, enabling structured database migrations.
- Created `database/migrations/env.py` to manage migration environment and database URL retrieval.
- Added migration script template `database/migrations/script.py.mako` for generating migration scripts.
- Updated `.gitignore` to exclude migration versions from version control.
- Enhanced `setup.md` documentation to include details on the migration system and commands for managing migrations.
- Introduced `database/redis_manager.py` to manage Redis connections, including synchronous and asynchronous clients.
- Implemented pub/sub messaging capabilities for real-time data distribution, with structured channel definitions for market data, bot signals, and system health.
- Added configuration options for Redis connection pooling and error handling, ensuring robust integration with the Crypto Trading Bot Platform.
