Simon/lowkey_backtest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add daily model training scripts and terminal UI for live trading

Browse Source 
- Introduced `train_daily.sh` for automating daily model retraining, including data download and model training steps.
- Added `install_cron.sh` for setting up a cron job to run the daily training script.
- Created `setup_schedule.sh` for configuring Systemd timers for daily training tasks.
- Implemented a terminal UI using Rich for real-time monitoring of trading performance, including metrics display and log handling.
- Updated `pyproject.toml` to include the `rich` dependency for UI functionality.
- Enhanced `.gitignore` to exclude model and log files.
- Added database support for trade persistence and metrics calculation.
- Updated README with installation and usage instructions for the new features.
This commit is contained in:
Simon Moisy
2026-01-18 11:08:57 +08:00
parent 35992ee374
commit b5550f4ff4
27 changed files with 3582 additions and 113 deletions
Download Patch File Download Diff File Expand all files Collapse all files

351
tasks/prd-terminal-ui.md Normal file
View File

@@ -0,0 +1,351 @@
# PRD: Terminal UI for Live Trading Bot
## Introduction/Overview
The live trading bot currently uses basic console logging for output, making it difficult to monitor trading activity, track performance, and understand the system state at a glance. This feature introduces a Rich-based terminal UI that provides a professional, real-time dashboard for monitoring the live trading bot.
The UI will display a horizontal split layout with a **summary panel** at the top (with tabbed time-period views) and a **scrollable log panel** at the bottom. The interface will update every second and support keyboard navigation.
## Goals
1. Provide real-time visibility into trading performance (PnL, win rate, trade count)
2. Enable monitoring of current position state (entry, SL/TP, unrealized PnL)
3. Display strategy signals (Z-score, model probability) for transparency
4. Support historical performance tracking across time periods (daily, weekly, monthly, all-time)
5. Improve operational experience with keyboard shortcuts and log filtering
6. Create a responsive design that works across different terminal sizes
## User Stories
1. **As a trader**, I want to see my total PnL and daily PnL at a glance so I can quickly assess performance.
2. **As a trader**, I want to see my current position details (entry price, unrealized PnL, SL/TP levels) so I can monitor risk.
3. **As a trader**, I want to view performance metrics by time period (daily, weekly, monthly) so I can track trends.
4. **As a trader**, I want to filter logs by type (errors, trades, signals) so I can focus on relevant information.
5. **As a trader**, I want keyboard shortcuts to navigate the UI without using a mouse.
6. **As a trader**, I want the UI to show strategy state (Z-score, probability) so I understand why signals are generated.
## Functional Requirements
### FR1: Layout Structure
1.1. The UI must use a horizontal split layout with the summary panel at the top and logs panel at the bottom.
1.2. The summary panel must contain tabbed views accessible via number keys:
- Tab 1 (`1`): **General** - Overall metrics since bot started
- Tab 2 (`2`): **Monthly** - Current month metrics (shown only if data spans > 1 month)
- Tab 3 (`3`): **Weekly** - Current week metrics (shown only if data spans > 1 week)
- Tab 4 (`4`): **Daily** - Today's metrics
1.3. The logs panel must be a scrollable area showing recent log entries.
1.4. The UI must be responsive and adapt to terminal size (minimum 80x24).
### FR2: Metrics Display
The summary panel must display the following metrics:
**Performance Metrics:**
2.1. Total PnL (USD) - cumulative profit/loss since tracking began
2.2. Period PnL (USD) - profit/loss for selected time period (daily/weekly/monthly)
2.3. Win Rate (%) - percentage of winning trades
2.4. Total Number of Trades
2.5. Average Trade Duration (hours)
2.6. Max Drawdown (USD and %)
**Current Position (if open):**
2.7. Symbol and side (long/short)
2.8. Entry price
2.9. Current price
2.10. Unrealized PnL (USD and %)
2.11. Stop-loss price and distance (%)
2.12. Take-profit price and distance (%)
2.13. Position size (USD)
**Account Status:**
2.14. Account balance / available margin (USDT)
2.15. Current leverage setting
**Strategy State:**
2.16. Current Z-score
2.17. Model probability
2.18. Current funding rate (BTC)
2.19. Last signal action and reason
### FR3: Historical Data Loading
3.1. On startup, the system must initialize SQLite database at `live_trading/trading.db`.
3.2. If `trade_log.csv` exists and database is empty, migrate CSV data to SQLite.
3.3. The UI must load current positions from `live_trading/positions.json` (kept for compatibility with existing position manager).
3.4. Metrics must be calculated via SQL aggregation queries for each time period.
3.5. If no historical data exists, the UI must show "No data" gracefully.
3.6. New trades must be written to both SQLite (primary) and CSV (backup/compatibility).
### FR4: Real-Time Updates
4.1. The UI must refresh every 1 second.
4.2. Position unrealized PnL must update based on latest price data.
4.3. New log entries must appear in real-time.
4.4. Metrics must recalculate when trades are opened/closed.
### FR5: Log Panel
5.1. The log panel must display log entries with timestamp, level, and message.
5.2. Log entries must be color-coded by level:
- ERROR: Red
- WARNING: Yellow
- INFO: White/Default
- DEBUG: Gray (if shown)
5.3. The log panel must support filtering by log type:
- All logs (default)
- Errors only
- Trades only (entries containing "position", "trade", "order")
- Signals only (entries containing "signal", "z_score", "prob")
5.4. Filter switching must be available via keyboard shortcut (`f` to cycle filters).
### FR6: Keyboard Controls
6.1. `q` or `Ctrl+C` - Graceful shutdown
6.2. `r` - Force refresh data
6.3. `1` - Switch to General tab
6.4. `2` - Switch to Monthly tab
6.5. `3` - Switch to Weekly tab
6.6. `4` - Switch to Daily tab
6.7. `f` - Cycle log filter
6.8. Arrow keys - Scroll logs (if supported)
### FR7: Color Scheme
7.1. Use dark theme as base.
7.2. PnL values must be colored:
- Positive: Green
- Negative: Red
- Zero/Neutral: White
7.3. Position side must be colored:
- Long: Green
- Short: Red
7.4. Use consistent color coding for emphasis and warnings.
## Non-Goals (Out of Scope)
1. **Mouse support** - This is a keyboard-driven terminal UI
2. **Trade execution from UI** - The UI is read-only; trades are executed by the bot
3. **Configuration editing** - Config changes require restarting the bot
4. **Multi-exchange support** - Only OKX is supported
5. **Charts/graphs** - Text-based metrics only (no ASCII charts in v1)
6. **Sound alerts** - No audio notifications
7. **Remote access** - Local terminal only
## Design Considerations
### Technology Choice: Rich
Use the [Rich](https://github.com/Textualize/rich) Python library for terminal UI:
- Rich provides `Live` display for real-time updates
- Rich `Layout` for split-screen design
- Rich `Table` for metrics display
- Rich `Panel` for bordered sections
- Rich `Text` for colored output
Alternative considered: **Textual** (also by Will McGugan) provides more advanced TUI features but adds complexity. Rich is simpler and sufficient for this use case.
### UI Mockup
```
+==============================================================================+
| REGIME REVERSION STRATEGY - LIVE TRADING [DEMO] ETH/USDT |
+==============================================================================+
| [1:General] [2:Monthly] [3:Weekly] [4:Daily] |
+------------------------------------------------------------------------------+
| PERFORMANCE | CURRENT POSITION |
| Total PnL: $1,234.56 | Side: LONG |
| Today PnL: $45.23 | Entry: $3,245.50 |
| Win Rate: 67.5% | Current: $3,289.00 |
| Total Trades: 24 | Unrealized: +$43.50 (+1.34%) |
| Avg Duration: 4.2h | Size: $500.00 |
| Max Drawdown: -$156.00 | SL: $3,050.00 (-6.0%) TP: $3,408.00 (+5%)|
| | |
| ACCOUNT | STRATEGY STATE |
| Balance: $5,432.10 | Z-Score: 1.45 |
| Available: $4,932.10 | Probability: 0.72 |
| Leverage: 2x | Funding: 0.0012 |
+------------------------------------------------------------------------------+
| LOGS [Filter: All] Press 'f' cycle |
+------------------------------------------------------------------------------+
| 14:32:15 [INFO] Trading Cycle Start: 2026-01-16T14:32:15+00:00 |
| 14:32:16 [INFO] Signal: entry long (prob=0.72, z=-1.45, reason=z_score...) |
| 14:32:17 [INFO] Executing LONG entry: 0.1540 ETH @ 3245.50 ($500.00) |
| 14:32:18 [INFO] Position opened: ETH/USDT:USDT_20260116_143217 |
| 14:32:18 [INFO] Portfolio: 1 positions, exposure=$500.00, unrealized=$0.00 |
| 14:32:18 [INFO] --- Cycle completed in 3.2s --- |
| 14:32:18 [INFO] Sleeping for 60 minutes... |
| |
+------------------------------------------------------------------------------+
| [q]Quit [r]Refresh [1-4]Tabs [f]Filter |
+==============================================================================+
```
### File Structure
```
live_trading/
ui/
__init__.py
dashboard.py # Main UI orchestration and threading
panels.py # Panel components (metrics, logs, position)
state.py # Thread-safe shared state
log_handler.py # Custom logging handler for UI queue
keyboard.py # Keyboard input handling
db/
__init__.py
database.py # SQLite connection and queries
models.py # Data models (Trade, DailySummary, Session)
migrations.py # CSV migration and schema setup
metrics.py # Metrics aggregation queries
trading.db # SQLite database file (created at runtime)
```
## Technical Considerations
### Integration with Existing Code
1. **Logging Integration**: Create a custom `logging.Handler` that captures log messages and forwards them to the UI log panel while still writing to file.
2. **Data Access**: The UI needs access to:
- `PositionManager` for current positions
- `TradingConfig` for settings display
- SQLite database for historical metrics
- Real-time data from `DataFeed` for current prices
3. **Main Loop Modification**: The `LiveTradingBot.run()` method needs modification to run the UI in a **separate thread** alongside the trading loop. The UI thread handles rendering and keyboard input while the main thread executes trading logic.
4. **Graceful Shutdown**: Ensure `SIGINT`/`SIGTERM` handlers work with the UI layer and properly terminate the UI thread.
### Database Schema (SQLite)
Create `live_trading/trading.db` with the following schema:
```sql
-- Trade history table
CREATE TABLE trades (
id INTEGER PRIMARY KEY AUTOINCREMENT,
trade_id TEXT UNIQUE NOT NULL,
symbol TEXT NOT NULL,
side TEXT NOT NULL, -- 'long' or 'short'
entry_price REAL NOT NULL,
exit_price REAL,
size REAL NOT NULL,
size_usdt REAL NOT NULL,
pnl_usd REAL,
pnl_pct REAL,
entry_time TEXT NOT NULL, -- ISO format
exit_time TEXT,
hold_duration_hours REAL,
reason TEXT, -- 'stop_loss', 'take_profit', 'signal', etc.
order_id_entry TEXT,
order_id_exit TEXT,
created_at TEXT DEFAULT CURRENT_TIMESTAMP
);
-- Daily summary table (for faster queries)
CREATE TABLE daily_summary (
id INTEGER PRIMARY KEY AUTOINCREMENT,
date TEXT UNIQUE NOT NULL, -- YYYY-MM-DD
total_trades INTEGER DEFAULT 0,
winning_trades INTEGER DEFAULT 0,
total_pnl_usd REAL DEFAULT 0,
max_drawdown_usd REAL DEFAULT 0,
updated_at TEXT DEFAULT CURRENT_TIMESTAMP
);
-- Session metadata
CREATE TABLE sessions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
start_time TEXT NOT NULL,
end_time TEXT,
starting_balance REAL,
ending_balance REAL,
total_pnl REAL,
total_trades INTEGER DEFAULT 0
);
-- Indexes for common queries
CREATE INDEX idx_trades_entry_time ON trades(entry_time);
CREATE INDEX idx_trades_exit_time ON trades(exit_time);
CREATE INDEX idx_daily_summary_date ON daily_summary(date);
```
### Migration from CSV
On first run with the new system:
1. Check if `trading.db` exists
2. If not, create database with schema
3. If `trade_log.csv` exists, migrate data to `trades` table
4. Rebuild `daily_summary` from migrated trades
### Dependencies
Add to `pyproject.toml`:
```toml
dependencies = [
# ... existing deps
"rich>=13.0.0",
]
```
Note: SQLite is part of Python's standard library (`sqlite3`), no additional dependency needed.
### Performance Considerations
- UI runs in a separate thread to avoid blocking trading logic
- Log buffer limited to 1000 entries in memory to prevent growth
- SQLite queries should use indexes for fast period-based aggregations
- Historical data loading happens once at startup, incremental updates thereafter
### Threading Model
```
Main Thread UI Thread
| |
v v
[Trading Loop] [Rich Live Display]
| |
+---> SharedState <------------+
(thread-safe)
| |
+---> LogQueue <---------------+
(thread-safe)
```
- Use `threading.Lock` for shared state access
- Use `queue.Queue` for log message passing
- UI thread polls for updates every 1 second
## Success Metrics
1. UI starts successfully and displays all required metrics
2. UI updates in real-time (1-second refresh) without impacting trading performance
3. All keyboard shortcuts function correctly
4. Historical data loads and displays accurately from SQLite
5. Log filtering works as expected
6. UI gracefully handles edge cases (no data, no position, terminal resize)
7. CSV migration completes successfully on first run
8. Database queries complete within 100ms
---
*Generated: 2026-01-16*
*Decisions: Threading model, 1000 log buffer, SQLite database, no fallback mode*