vasily/TCPDashboard
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

dev setup

Browse Source
This commit is contained in:
Ajasra
2025-05-29 23:50:41 +08:00
parent d90681b788
commit f5b8bd3c96
17 changed files with 1462 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files

259
docs/dependency-management.md Normal file
View File

@@ -0,0 +1,259 @@
# Dependency Management Guide
This guide explains how to manage Python dependencies in the Crypto Trading Bot Dashboard project.
## Local Development
### Adding New Dependencies
#### 1. Core Dependencies (Required for Runtime)
To add a new core dependency:
```bash
# Method 1: Add directly to pyproject.toml
# Edit pyproject.toml and add to the dependencies list:
# "new-package>=1.0.0",
# Method 2: Use UV to add and update pyproject.toml
uv add "new-package>=1.0.0"
# Sync to install
uv sync
```
#### 2. Development Dependencies (Testing, Linting, etc.)
```bash
# Add development-only dependency
uv add --dev "new-dev-package>=1.0.0"
# Or edit pyproject.toml under [project.optional-dependencies.dev]
# Then run:
uv sync --dev
```
### Installing Dependencies
```bash
# Install all dependencies
uv sync
# Install with development dependencies
uv sync --dev
# Install only production dependencies
uv sync --no-dev
```
### Updating Dependencies
```bash
# Update all dependencies to latest compatible versions
uv sync --upgrade
# Update specific package
uv sync --upgrade-package "package-name"
```
## Docker Environment
### Current Approach
The project uses a **volume-based development** approach where:
- Dependencies are installed in the local environment using UV
- Docker containers provide only infrastructure services (PostgreSQL, Redis)
- The Python application runs locally with hot reload
### Adding Dependencies for Docker-based Development
If you want to run the entire application in Docker:
#### 1. Create a Dockerfile
```dockerfile
FROM python:3.10-slim
WORKDIR /app
# Install system dependencies
RUN apt-get update && apt-get install -y \
postgresql-client \
curl \
&& rm -rf /var/lib/apt/lists/*
# Install UV
RUN pip install uv
# Copy dependency files
COPY pyproject.toml ./
COPY README.md ./
# Install dependencies
RUN uv sync --no-dev
# Copy application code
COPY . .
# Expose port
EXPOSE 8050
# Run application
CMD ["uv", "run", "python", "main.py"]
```
#### 2. Add Application Service to docker-compose.yml
```yaml
services:
app:
build: .
container_name: dashboard_app
ports:
- "8050:8050"
volumes:
- .:/app
- uv_cache:/root/.cache/uv
environment:
- DATABASE_URL=postgresql://dashboard:dashboard123@postgres:5432/dashboard
- REDIS_URL=redis://redis:6379
depends_on:
- postgres
- redis
networks:
- dashboard-network
restart: unless-stopped
volumes:
uv_cache:
```
#### 3. Development Workflow with Docker
```bash
# Build and start all services
docker-compose up --build
# Add new dependency
# 1. Edit pyproject.toml
# 2. Rebuild container
docker-compose build app
docker-compose up -d app
# Or use dev dependencies mount
# Mount local UV cache for faster rebuilds
```
## Hot Reload Development
### Method 1: Local Development (Recommended)
Run services in Docker, application locally with hot reload:
```bash
# Start infrastructure
python scripts/dev.py start
# Run app with hot reload
uv run python scripts/dev.py dev-server
```
### Method 2: Docker with Volume Mounts
If using Docker for the app, mount source code:
```yaml
volumes:
- .:/app # Mount source code
- /app/__pycache__ # Exclude cache
```
## Best Practices
### 1. Version Pinning
```toml
# Good: Specify minimum version with compatibility
"requests>=2.31.0,<3.0.0"
# Acceptable: Major version constraint
"pandas>=2.1.0"
# Avoid: Exact pinning (except for critical deps)
"somepackage==1.2.3" # Only if necessary
```
### 2. Dependency Categories
```toml
[project]
dependencies = [
# Core web framework
"dash>=2.14.0",
# Database
"sqlalchemy>=2.0.0",
"psycopg2-binary>=2.9.0",
# ... group related dependencies
]
```
### 3. Security Updates
```bash
# Check for security vulnerabilities
pip-audit
# Update specific vulnerable package
uv sync --upgrade-package "vulnerable-package"
```
## Troubleshooting
### Common Issues
1. **Dependency Conflicts**
```bash
# Clear UV cache and reinstall
uv cache clean
uv sync --refresh
```
2. **PostgreSQL Connection Issues**
```bash
# Ensure psycopg2-binary is installed
uv add "psycopg2-binary>=2.9.0"
```
3. **Docker Build Failures**
```bash
# Clean docker build cache
docker system prune --volumes
docker-compose build --no-cache
```
### Debugging Dependencies
```bash
# Show installed packages
uv pip list
# Show dependency tree
uv pip show <package-name>
# Check for conflicts
uv pip check
```
## Migration from requirements.txt
If you have an existing `requirements.txt`:
```bash
# Convert to pyproject.toml
uv add -r requirements.txt
# Or manually copy dependencies to pyproject.toml
# Then remove requirements.txt
```

282
docs/setup.md Normal file
View File

@@ -0,0 +1,282 @@
# Development Environment Setup
This guide will help you set up the Crypto Trading Bot Dashboard development environment.
## Prerequisites
- Python 3.10+
- Docker Desktop (for Windows/Mac) or Docker Engine (for Linux)
- UV package manager
- Git
## Quick Start
### 1. Initial Setup
```bash
# Install dependencies (including dev tools)
uv sync --dev
# Set up environment and start services
python scripts/dev.py setup
```
### 2. Start Services
```bash
# Start PostgreSQL and Redis services
python scripts/dev.py start
```
### 3. Configure API Keys
Copy `env.template` to `.env` and update the OKX API credentials:
```bash
# Copy template (Windows)
copy env.template .env
# Copy template (Unix)
cp env.template .env
# Edit .env file with your actual OKX API credentials
# OKX_API_KEY=your_actual_api_key
# OKX_SECRET_KEY=your_actual_secret_key
# OKX_PASSPHRASE=your_actual_passphrase
```
### 4. Verify Setup
```bash
# Run setup verification tests
uv run python tests/test_setup.py
```
### 5. Start Dashboard with Hot Reload
```bash
# Start with hot reload (recommended for development)
python scripts/dev.py dev-server
# Or start without hot reload
python scripts/dev.py run
```
## Development Commands
### Using the dev.py script:
```bash
# Show available commands
python scripts/dev.py
# Set up environment and install dependencies
python scripts/dev.py setup
# Start all services (Docker)
python scripts/dev.py start
# Stop all services
python scripts/dev.py stop
# Restart services
python scripts/dev.py restart
# Check service status
python scripts/dev.py status
# Install/update dependencies
python scripts/dev.py install
# Run development server with hot reload
python scripts/dev.py dev-server
# Run application without hot reload
python scripts/dev.py run
```
### Direct Docker commands:
```bash
# Start services in background
docker-compose up -d
# View service logs
docker-compose logs -f
# Stop services
docker-compose down
# Rebuild and restart
docker-compose up -d --build
```
## Hot Reload Development
The development server includes hot reload functionality that automatically restarts the application when Python files change.
### Features:
- 🔥 **Auto-restart** on file changes
- 👀 **Watches multiple directories** (config, database, components, etc.)
- 🚀 **Fast restart** with debouncing (1-second delay)
- 🛑 **Graceful shutdown** with Ctrl+C
### Usage:
```bash
# Start hot reload server
python scripts/dev.py dev-server
# The server will watch these directories:
# - . (root)
# - config/
# - database/
# - components/
# - data/
# - strategies/
# - trader/
```
## Data Persistence
### Database Persistence
✅ **PostgreSQL data persists** across container restarts
- Volume: `postgres_data` mounted to `/var/lib/postgresql/data`
- Data survives `docker-compose down` and `docker-compose up`
### Redis Persistence
✅ **Redis data persists** with AOF (Append-Only File)
- Volume: `redis_data` mounted to `/data`
- AOF sync every second for durability
- Data survives container restarts
### Removing Persistent Data
```bash
# Stop services and remove volumes (CAUTION: This deletes all data)
docker-compose down -v
# Or remove specific volumes
docker volume rm dashboard_postgres_data
docker volume rm dashboard_redis_data
```
## Dependency Management
### Adding New Dependencies
```bash
# Add runtime dependency
uv add "package-name>=1.0.0"
# Add development dependency
uv add --dev "dev-package>=1.0.0"
# Install all dependencies
uv sync --dev
```
### Key Dependencies Included:
- **Web Framework**: Dash, Plotly
- **Database**: SQLAlchemy, psycopg2-binary, Alembic
- **Data Processing**: pandas, numpy
- **Configuration**: pydantic, python-dotenv
- **Development**: watchdog (hot reload), pytest, black, mypy
See `docs/dependency-management.md` for detailed dependency management guide.
## Directory Structure
```
Dashboard/
├── config/ # Configuration files
│ ├── settings.py # Application settings
│ └── bot_configs/ # Bot configuration files
├── database/ # Database related files
│ └── init/ # Database initialization scripts
├── scripts/ # Development scripts
│ ├── dev.py # Main development script
│ ├── setup.sh # Setup script (Unix)
│ ├── start.sh # Start script (Unix)
│ └── stop.sh # Stop script (Unix)
├── tests/ # Test files
│ └── test_setup.py # Setup verification tests
├── docs/ # Documentation
│ ├── setup.md # This file
│ └── dependency-management.md # Dependency guide
├── docker-compose.yml # Docker services configuration
├── env.template # Environment variables template
├── pyproject.toml # Dependencies and project config
└── main.py # Main application entry point
```
## Services
### PostgreSQL Database
- **Host**: localhost:5432
- **Database**: dashboard
- **User**: dashboard
- **Password**: dashboard123 (development only)
- **Persistence**: ✅ Data persists across restarts
### Redis Cache
- **Host**: localhost:6379
- **No password** (development only)
- **Persistence**: ✅ AOF enabled, data persists across restarts
## Environment Variables
Key environment variables (see `env.template` for full list):
- `DATABASE_URL` - PostgreSQL connection string
- `OKX_API_KEY` - OKX API key
- `OKX_SECRET_KEY` - OKX secret key
- `OKX_PASSPHRASE` - OKX passphrase
- `OKX_SANDBOX` - Use OKX sandbox (true/false)
- `DEBUG` - Enable debug mode
- `LOG_LEVEL` - Logging level (DEBUG, INFO, WARNING, ERROR)
## Troubleshooting
### Docker Issues
1. **Docker not running**: Start Docker Desktop/Engine
2. **Port conflicts**: Check if ports 5432 or 6379 are already in use
3. **Permission issues**: On Linux, add your user to the docker group
4. **Data persistence issues**: Check if volumes are properly mounted
### Database Connection Issues
1. **Connection refused**: Ensure PostgreSQL container is running
2. **Authentication failed**: Check credentials in `.env` file
3. **Database doesn't exist**: Run the setup script again
4. **Data loss**: Check if volume is mounted correctly
### Dependency Issues
1. **Import errors**: Run `uv sync --dev` to install dependencies
2. **Version conflicts**: Check `pyproject.toml` for compatibility
3. **Hot reload not working**: Ensure `watchdog` is installed
### Hot Reload Issues
1. **Changes not detected**: Check if files are in watched directories
2. **Rapid restarts**: Built-in 1-second debouncing should prevent this
3. **Process not stopping**: Use Ctrl+C to gracefully shutdown
## Performance Tips
1. **Use SSD**: Store Docker volumes on SSD for better database performance
2. **Increase Docker memory**: Allocate more RAM to Docker Desktop
3. **Hot reload**: Use `dev-server` for faster development cycles
4. **Dependency caching**: UV caches dependencies for faster installs
## Next Steps
After successful setup:
1. **Phase 1.0**: Database Infrastructure Setup
2. **Phase 2.0**: Bot Management System Development
3. **Phase 3.0**: OKX Integration and Data Pipeline
4. **Phase 4.0**: Dashboard UI and Visualization
5. **Phase 5.0**: Backtesting System Implementation
See `tasks/tasks-prd-crypto-bot-dashboard.md` for detailed task list.