3.5 Enhance system health monitoring dashboard with comprehensive market data tracking

- Added `psutil` dependency for system performance metrics. - Implemented a new layout in `dashboard/layouts/system_health.py` using Mantine components for real-time monitoring of data collection services, database health, Redis status, and system performance. - Enhanced callbacks in `dashboard/callbacks/system_health.py` for detailed status updates and error handling. - Introduced quick status indicators for data collection, database, Redis, and performance metrics with auto-refresh functionality. - Created modals for viewing detailed data collection information and service logs. - Updated documentation to reflect the new features and usage guidelines.
2025-06-04 17:46:50 +08:00
parent 8aa47731f2
commit 82f4e0ef48
7 changed files with 1097 additions and 119 deletions
--- a/Dashboard.md
+++ b/Dashboard.md
@@ -0,0 +1,205 @@
+# Task 3.5 - Market Data Monitoring Dashboard
+
+**Status**: ✅ **COMPLETED**
+
+## Overview
+Implemented a comprehensive market data monitoring dashboard with real-time data feed status monitoring, database health tracking, Redis monitoring, and system performance metrics.
+
+## Implementation Details
+
+### Key Features Implemented
+
+1. **Real-time Status Overview**
+   - Quick status cards for Data Collection, Database, Redis, and Performance
+   - Color-coded badges (green/yellow/red) for instant status recognition
+   - Auto-refreshing status indicators every 30 seconds
+
+2. **Data Collection Service Monitoring**
+   - Service running status detection
+   - Data collection metrics (candles, tickers collected)
+   - Data freshness indicators
+   - Service control buttons (refresh, view details, view logs)
+
+3. **Individual Collectors Health**
+   - Placeholder for collector health monitoring
+   - Ready for integration with data collection service health API
+   - Instructions for starting monitoring
+
+4. **Database Health Monitoring**
+   - Connection status verification
+   - PostgreSQL version and connection count
+   - Database statistics (table sizes, recent activity)
+   - Performance metrics
+
+5. **Redis Status Monitoring**
+   - Connection verification
+   - Redis server information
+   - Memory usage and client statistics
+   - Uptime tracking
+
+6. **System Performance Metrics**
+   - CPU usage with color-coded warnings
+   - Memory utilization
+   - Disk usage monitoring
+   - Network I/O statistics
+
+7. **Interactive Features**
+   - Data collection details modal
+   - Service logs viewer modal
+   - Refresh controls for real-time updates
+
+### UI Framework
+- **Mantine Components**: Used Mantine UI library for consistency with existing dashboard
+- **Responsive Layout**: Grid-based layout for optimal viewing
+- **Modern Design**: Cards, badges, alerts, and modals for professional appearance
+
+### Files Modified/Created
+
+1. **`dashboard/layouts/system_health.py`**
+   - Complete rewrite using Mantine components
+   - Comprehensive layout with monitoring sections
+   - Modal dialogs for detailed views
+
+2. **`dashboard/callbacks/system_health.py`**
+   - Enhanced callbacks with comprehensive monitoring
+   - Real-time status updates
+   - Error handling and graceful degradation
+   - Integration with database and Redis managers
+
+## Technical Implementation
+
+### Real-time Monitoring Architecture
+```python
+# Status Update Flow
+Interval Component (30s) → Callbacks → Status Checkers → UI Updates
+```
+
+### Status Checking Functions
+- `_get_data_collection_quick_status()` - Service running detection
+- `_get_database_quick_status()` - Database connectivity
+- `_get_redis_quick_status()` - Redis connectivity  
+- `_get_performance_quick_status()` - System metrics
+
+### Detailed Monitoring Functions
+- `_get_data_collection_service_status()` - Service details
+- `_get_data_collection_metrics()` - Collection statistics
+- `_get_database_status()` & `_get_database_statistics()` - DB health
+- `_get_redis_status()` & `_get_redis_statistics()` - Redis health
+- `_get_system_performance_metrics()` - System performance
+
+### Error Handling
+- Graceful degradation when services are unavailable
+- User-friendly error messages with troubleshooting hints
+- Fallback status indicators for unknown states
+
+## Integration Points
+
+### Database Integration
+- Uses `DatabaseManager` for connection testing
+- Queries `market_data` table for collection statistics
+- Monitors database performance metrics
+
+### Redis Integration  
+- Uses `RedisManager` for connection verification
+- Retrieves Redis server information and statistics
+- Monitors memory usage and client connections
+
+### System Integration
+- Uses `psutil` for system performance monitoring
+- Process detection for data collection service
+- Resource utilization tracking
+
+## Usage
+
+### Dashboard Access
+1. Navigate to "⚙️ System Health" tab in the main dashboard
+2. View real-time status cards at the top
+3. Explore detailed monitoring sections below
+
+### Service Controls
+- **Refresh Status**: Manually refresh data collection status
+- **View Details**: Open modal with comprehensive service information
+- **View Logs**: Access service logs in scrollable modal
+
+### Status Indicators
+- 🟢 **Green**: Healthy/Connected/Good performance
+- 🟡 **Yellow**: Warning/Checking/Moderate usage
+- 🔴 **Red**: Error/Disconnected/High usage
+- ❓ **Gray**: Unknown status
+
+## Future Enhancements
+
+### Planned Improvements (Section 3.7)
+1. **Real-time Updates via Redis**: Replace polling with Redis pub/sub
+2. **Advanced Metrics**: Historical performance trends
+3. **Alerting System**: Notifications for critical issues
+4. **Service Management**: Start/stop controls for data collection
+
+### Integration with Data Collection Service
+- Real-time collector health reporting
+- Performance metrics streaming
+- Service configuration management
+- Log aggregation and filtering
+
+## Testing
+
+### Manual Testing
+1. **Service Detection**: Start/stop data collection service to verify detection
+2. **Database Connectivity**: Test with database running/stopped
+3. **Redis Connectivity**: Test with Redis running/stopped
+4. **Performance Monitoring**: Verify metrics under different system loads
+
+### Integration Testing
+- Database manager integration
+- Redis manager integration  
+- System metrics accuracy
+- Error handling scenarios
+
+## Dependencies
+
+### UI Framework
+- `dash-mantine-components` - Modern UI components
+- `dash` - Core dashboard framework
+- `plotly` - Charts and visualizations
+
+### System Monitoring
+- `psutil` - System performance metrics
+- `subprocess` - Process management
+- `datetime` - Time handling
+
+### Database/Redis
+- `database.connection.DatabaseManager` - Database operations
+- `database.redis_manager.RedisManager` - Redis operations
+
+## Troubleshooting
+
+### Common Issues
+
+1. **"Service Stopped" Status**
+   - Solution: Run `python scripts/start_data_collection.py`
+
+2. **Database Connection Failed**
+   - Check Docker containers: `docker-compose ps`
+   - Verify database configuration in `.env`
+
+3. **Redis Connection Failed**
+   - Ensure Redis container is running
+   - Check Redis configuration
+
+4. **Performance Metrics Unavailable**
+   - Usually permissions issue on system metrics
+   - Check if `psutil` has necessary permissions
+
+### Logs and Debugging
+- Check dashboard logs for callback errors
+- Use browser developer tools for frontend issues
+- Monitor system logs for resource issues
+
+## Documentation Updates
+
+### Files Updated
+- `tasks/tasks-crypto-bot-prd.md` - Marked Task 3.5 as completed
+- Added this documentation file
+
+### Next Task
+Ready to proceed with **Task 3.6**: Build simple data analysis tools (volume analysis, price movement statistics) 
--- a/tasks/tasks-crypto-bot-prd.md
+++ b/tasks/tasks-crypto-bot-prd.md
@@ -48,6 +48,8 @@
 - `docs/logging.md` - Complete documentation for the enhanced unified logging system
 - `docs/data-collection-service.md` - Complete documentation for the data collection service with usage examples, configuration, and deployment guide
 - `docs/components/technical-indicators.md` - Complete documentation for the technical indicators module with usage examples and integration guide
+- `dashboard/layouts/system_health.py` - Enhanced system health monitoring layout with comprehensive market data monitoring using Mantine components
+- `dashboard/callbacks/system_health.py` - Enhanced system health callbacks with real-time data collection monitoring, database statistics, Redis monitoring, and performance metrics using Mantine components

 ## Tasks

@@ -80,8 +82,8 @@
  - [x] 3.1 Setup Dash application framework with Mantine UI components
  - [x] 3.2 Create basic layout and navigation structure
  - [x] 3.3 Implement real-time OHLCV price charts with Plotly (candlestick charts)
-  - [ ] 3.4 Add technical indicators overlay on price charts (SMA, EMA, RSI, MACD)
-  - [ ] 3.5 Create market data monitoring dashboard (real-time data feed status)
+  - [x] 3.4 Add technical indicators overlay on price charts (SMA, EMA, RSI, MACD)
+  - [x] 3.5 Create market data monitoring dashboard (real-time data feed status)
  - [ ] 3.6 Build simple data analysis tools (volume analysis, price movement statistics)
  - [ ] 3.7 Setup real-time dashboard updates using Redis callbacks
  - [ ] 3.8 Add data export functionality for analysis (CSV/JSON export)