vasily/TCPDashboard
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor database configuration and schema for Crypto Trading Bot Platform

Browse Source 
- Updated `docker-compose.yml` to remove hardcoded passwords, relying on environment variables for PostgreSQL and Redis configurations.
- Modified `env.template` to reflect new password settings and ensure secure handling of sensitive information.
- Introduced a new `database/connection.py` file for improved database connection management, including connection pooling and session handling.
- Updated `database/models.py` to align with the new schema in `schema_clean.sql`, utilizing JSONB for optimized data storage.
- Enhanced `setup.md` documentation to clarify the initialization process and emphasize the importance of the `.env` file for configuration.
- Added a new `scripts/init_database.py` script for automated database initialization and verification, ensuring all tables are created as expected.
This commit is contained in:
Vasily.onl
2025-05-30 18:20:38 +08:00
parent 8121ce0430
commit 73b7e8bb9d
12 changed files with 781 additions and 142 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
tasks/tasks-crypto-bot-prd.md
View File

@@ -2,9 +2,12 @@
- `app.py` - Main Dash application entry point and dashboard interface
- `bot_manager.py` - Bot lifecycle management and coordination
- `database/models.py` - PostgreSQL database models and schema definitions
- `database/schema.sql` - Complete database schema with all tables, indexes, and constraints
- `database/connection.py` - Database connection and query utilities
- `database/models.py` - PostgreSQL database models and schema definitions (updated to match schema_clean.sql)
- `database/schema_clean.sql` - Clean database schema without hypertables (actively used, includes raw_trades table)
- `database/schema.sql` - Complete database schema with TimescaleDB hypertables (for future optimization)
- `database/connection.py` - Database connection utility with connection pooling, session management, and raw data utilities
- `database/init/init.sql` - Docker initialization script for automatic database setup
- `database/init/schema_clean.sql` - Copy of clean schema for Docker initialization
- `data/okx_collector.py` - OKX API integration for real-time market data collection
- `data/aggregator.py` - OHLCV candle aggregation and processing
- `strategies/base_strategy.py` - Base strategy class and interface
@@ -15,7 +18,9 @@
- `backtesting/performance.py` - Performance metrics calculation
- `config/bot_configs/` - Directory for JSON bot configuration files
- `config/strategies/` - Directory for JSON strategy parameter files
- `config/settings.py` - Centralized configuration settings using Pydantic
- `scripts/dev.py` - Development setup and management script
- `scripts/init_database.py` - Database initialization and verification script
- `requirements.txt` - Python dependencies managed by UV
- `docker-compose.yml` - Docker services configuration with TimescaleDB support
- `tests/test_strategies.py` - Unit tests for strategy implementations
@@ -23,19 +28,12 @@
- `tests/test_data_collection.py` - Unit tests for data collection and aggregation
- `docs/setup.md` - Comprehensive setup guide for new machines and environments
### Notes
- Unit tests should be placed in the `tests/` directory with descriptive names
- Use `uv run pytest` to run all tests or `uv run pytest tests/specific_test.py` for individual test files
- JSON configuration files allow rapid strategy parameter testing without code changes
- Redis will be used for real-time messaging between components
## Tasks
- [ ] 1.0 Database Foundation and Schema Setup
- [x] 1.1 Install and configure PostgreSQL with Docker
- [x] 1.2 Create database schema following the PRD specifications (market_data, bots, signals, trades, bot_performance tables)
- [ ] 1.3 Implement database connection utility with connection pooling
- [x] 1.3 Implement database connection utility with connection pooling
- [ ] 1.4 Create database models using SQLAlchemy or similar ORM
- [x] 1.5 Add proper indexes for time-series data optimization
- [ ] 1.6 Setup Redis for pub/sub messaging
@@ -161,3 +159,18 @@
- [ ] 13.7 Implement horizontal scaling for high-volume trading scenarios
### Notes
- **Automatic Database Setup**: Database schema is automatically initialized when Docker containers start via `database/init/` scripts
- **Environment Configuration**: All credentials and settings are managed via `.env` file with consistent defaults
- **Security**: No hardcoded passwords exist in the codebase - all credentials must be loaded from environment variables
- **Clean Schema Approach**: Using `schema_clean.sql` for simpler setup without TimescaleDB hypertables (can be upgraded later)
- Unit tests should be placed in the `tests/` directory with descriptive names
- Use `uv run pytest` to run all tests or `uv run pytest tests/specific_test.py` for individual test files
- JSON configuration files allow rapid strategy parameter testing without code changes
- Redis will be used for real-time messaging between components
- Database models now use JSONB instead of JSON for PostgreSQL optimization
- Connection pooling is configured with proper retry logic and monitoring
- Raw data is stored in PostgreSQL with automatic cleanup utilities (configurable retention period)
- Raw data storage includes: ticker data, trade data, orderbook snapshots, candle data, and balance updates