- Added `realtime_execution.py` for real-time strategy execution, enabling live signal generation and integration with the dashboard's chart refresh cycle.
- Introduced `data_integration.py` to manage market data orchestration, caching, and technical indicator calculations for strategy signal generation.
- Implemented `validation.py` for comprehensive validation and quality assessment of strategy-generated signals, ensuring reliability and consistency.
- Developed `batch_processing.py` to facilitate efficient backtesting of multiple strategies across large datasets with memory management and performance optimization.
- Updated `__init__.py` files to include new modules and ensure proper exports, enhancing modularity and maintainability.
- Enhanced unit tests for the new features, ensuring robust functionality and adherence to project standards.
These changes establish a solid foundation for real-time strategy execution and data integration, aligning with project goals for modularity, performance, and maintainability.
- Added `StrategyRun` and `StrategySignal` models to track strategy execution sessions and generated signals, respectively, ensuring a clear separation from live trading data.
- Introduced `StrategyRepository` for managing database operations related to strategy runs and signals, including methods for creating, updating, and retrieving strategy data.
- Updated `DatabaseOperations` to integrate the new repository, enhancing the overall architecture and maintaining consistency with existing database access patterns.
- Enhanced documentation to reflect the new database schema and repository functionalities, ensuring clarity for future development and usage.
These changes establish a robust foundation for strategy analysis and backtesting, aligning with project goals for modularity, performance, and maintainability.
- Deleted `app_new.py`, which was previously the main entry point for the dashboard application, to streamline the codebase.
- Consolidated the application initialization and callback registration logic into `main.py`, enhancing modularity and maintainability.
- Updated the logging and error handling practices in `main.py` to ensure consistent application behavior and improved debugging capabilities.
These changes simplify the application structure, aligning with project standards for modularity and maintainability.
- Introduced a comprehensive data collection framework, including `CollectorServiceConfig`, `BaseDataCollector`, and `CollectorManager`, enhancing modularity and maintainability.
- Developed `CollectorFactory` for streamlined collector creation, promoting separation of concerns and improved configuration handling.
- Enhanced `DataCollectionService` to utilize the new architecture, ensuring robust error handling and logging practices.
- Added `TaskManager` for efficient management of asynchronous tasks, improving performance and resource management.
- Implemented health monitoring and auto-recovery features in `CollectorManager`, ensuring reliable operation of data collectors.
- Updated imports across the codebase to reflect the new structure, ensuring consistent access to components.
These changes significantly improve the architecture and maintainability of the data collection service, aligning with project standards for modularity, performance, and error handling.
- Replaced the `RedisManager` class with a more modular `SyncRedisManager` and `AsyncRedisManager`, improving the separation of synchronous and asynchronous operations.
- Updated the `system_health.py` callbacks to utilize the new `get_sync_redis_manager` function for Redis interactions, simplifying the connection process.
- Enhanced error handling and logging in Redis status checks, providing clearer feedback on connection issues.
- Revised the setup documentation to reflect changes in Redis connection testing, ensuring clarity for users.
These updates improve the maintainability and reliability of Redis interactions within the system, aligning with best practices for modular design.
- Removed the `RawDataManager` class and integrated its functionality directly into the `RawTradeRepository`, streamlining the management of raw trade data.
- Implemented the `cleanup_old_raw_data` method to delete outdated records, preventing table bloat and improving performance.
- Added the `get_raw_data_stats` method to retrieve statistics about raw data storage, enhancing data management capabilities.
- Updated documentation to reflect the new methods and their usage, ensuring clarity for future developers.
These changes improve the maintainability and efficiency of the database operations related to raw trade data.
- Updated the `MarketDataRepository` and `RawTradeRepository` classes to exclusively utilize SQLAlchemy ORM for all database interactions, enhancing maintainability and type safety.
- Removed raw SQL queries in favor of ORM methods, ensuring a consistent and database-agnostic approach across the repository layer.
- Revised documentation to reflect these changes, emphasizing the importance of using the ORM for database operations.
These modifications improve the overall architecture of the database layer, making it more scalable and easier to manage.
- Introduced a modular repository structure by creating separate repository classes for `Bot`, `MarketData`, and `RawTrade`, improving code organization and maintainability.
- Updated the `DatabaseOperations` class to utilize the new repository classes, enhancing the abstraction of database interactions.
- Refactored the `.env` file to update database connection parameters and add new logging and health monitoring configurations.
- Modified the `okx_config.json` to change default timeframes for trading pairs, aligning with updated requirements.
- Added comprehensive unit tests for the new repository classes, ensuring robust functionality and reliability.
These changes improve the overall architecture of the database layer, making it more scalable and easier to manage.
- Suppressed SQLAlchemy logging in `app.py` and `main.py` to reduce console verbosity.
- Introduced a new modular chart system in `components/charts/` with a `ChartBuilder` class for flexible chart creation.
- Added utility functions for data processing and validation in `components/charts/utils.py`.
- Implemented indicator definitions and configurations in `components/charts/config/indicator_defs.py`.
- Created a comprehensive documentation structure for the new chart system, ensuring clarity and maintainability.
- Added unit tests for the `ChartBuilder` class to verify functionality and robustness.
- Updated existing components to integrate with the new chart system, enhancing overall architecture and user experience.
- Introduced `app.py` as the main entry point for the dashboard, providing real-time visualization and bot management interface.
- Implemented layout components including header, navigation tabs, and content areas for market data, bot management, performance analytics, and system health.
- Added callbacks for dynamic updates of market data charts and statistics, ensuring real-time interaction.
- Created reusable UI components in `components` directory for modularity and maintainability.
- Enhanced database operations for fetching market data and checking data availability.
- Updated `main.py` to start the dashboard application with improved user instructions and error handling.
- Documented components and functions for clarity and future reference.
- Deleted `example_complete_series_aggregation.py` as it is no longer needed.
- Introduced `data_collection_service.py`, a production-ready service for cryptocurrency market data collection with clean logging and robust error handling.
- Added configuration management for multiple trading pairs and exchanges, supporting health monitoring and graceful shutdown.
- Created `data_collection.json` for service configuration, including exchange settings and logging preferences.
- Updated `CandleProcessingConfig` to reflect changes in timeframes for candle processing.
- Enhanced documentation to cover the new data collection service and its configuration, ensuring clarity for users.
- Introduced `example_complete_series_aggregation.py` to demonstrate time series aggregation, emitting candles even when no trades occur.
- Implemented `CompleteSeriesProcessor` extending `RealTimeCandleProcessor` to handle time-based candle emission and empty candle creation.
- Refactored `OKXCollector` to utilize the new repository pattern for database operations, enhancing modularity and maintainability.
- Updated database operations to centralize data handling through `DatabaseOperations`, improving error handling and logging.
- Enhanced documentation to include details on the new aggregation example and repository pattern implementation, ensuring clarity for users.
- Introduced `force_update_candles` option in `okx_config.json` to control candle update behavior.
- Updated `OKXCollector` to handle candle storage based on the `force_update_candles` setting, allowing for either updating existing records or preserving them.
- Enhanced logging to reflect the action taken during candle storage, improving traceability.
- Updated database schema to include `updated_at` timestamp for better tracking of data changes.
- Increased health check interval from 30s to 120s in `okx_config.json`.
- Added support for additional timeframes (1s, 5s, 10s, 15s, 30s) in the aggregation logic across multiple components.
- Updated `CandleProcessingConfig` and `RealTimeCandleProcessor` to handle new timeframes.
- Enhanced validation and parsing functions to include new second-based timeframes.
- Updated database schema to support new timeframes in `schema_clean.sql`.
- Improved documentation to reflect changes in multi-timeframe aggregation capabilities.
- Introduced `alembic.ini` for Alembic configuration, enabling structured database migrations.
- Created `database/migrations/env.py` to manage migration environment and database URL retrieval.
- Added migration script template `database/migrations/script.py.mako` for generating migration scripts.
- Updated `.gitignore` to exclude migration versions from version control.
- Enhanced `setup.md` documentation to include details on the migration system and commands for managing migrations.
- Introduced `database/redis_manager.py` to manage Redis connections, including synchronous and asynchronous clients.
- Implemented pub/sub messaging capabilities for real-time data distribution, with structured channel definitions for market data, bot signals, and system health.
- Added configuration options for Redis connection pooling and error handling, ensuring robust integration with the Crypto Trading Bot Platform.
- Updated `docker-compose.yml` to remove hardcoded passwords, relying on environment variables for PostgreSQL and Redis configurations.
- Modified `env.template` to reflect new password settings and ensure secure handling of sensitive information.
- Introduced a new `database/connection.py` file for improved database connection management, including connection pooling and session handling.
- Updated `database/models.py` to align with the new schema in `schema_clean.sql`, utilizing JSONB for optimized data storage.
- Enhanced `setup.md` documentation to clarify the initialization process and emphasize the importance of the `.env` file for configuration.
- Added a new `scripts/init_database.py` script for automated database initialization and verification, ensuring all tables are created as expected.
- Added new SQLAlchemy models in `database/models.py` for market data, trades, bots, signals, and performance tracking.
- Updated `docker-compose.yml` to use TimescaleDB for PostgreSQL and configured shared preload libraries.
- Created new schema files: `schema.sql` for the complete database setup and `schema_clean.sql` for a simplified version without hypertables.
- Updated documentation in `setup.md` to reflect changes in database initialization and service setup.
