TCPDashboard/tasks/3.5. Market Data Monitoring Dashboard.md

# Task 3.5 - Market Data Monitoring Dashboard

**Status**: ✅ **COMPLETED**

## Overview
Implemented a comprehensive market data monitoring dashboard with real-time data feed status monitoring, database health tracking, Redis monitoring, and system performance metrics.

## Implementation Details

### Key Features Implemented

1. **Real-time Status Overview**
   - Quick status cards for Data Collection, Database, Redis, and Performance
   - Color-coded badges (green/yellow/red) for instant status recognition
   - Auto-refreshing status indicators every 30 seconds

2. **Data Collection Service Monitoring**
   - Service running status detection
   - Data collection metrics (candles, tickers collected)
   - Data freshness indicators
   - Service control buttons (refresh, view details, view logs)

3. **Individual Collectors Health**
   - Placeholder for collector health monitoring
   - Ready for integration with data collection service health API
   - Instructions for starting monitoring

4. **Database Health Monitoring**
   - Connection status verification
   - PostgreSQL version and connection count
   - Database statistics (table sizes, recent activity)
   - Performance metrics

5. **Redis Status Monitoring**
   - Connection verification
   - Redis server information
   - Memory usage and client statistics
   - Uptime tracking

6. **System Performance Metrics**
   - CPU usage with color-coded warnings
   - Memory utilization
   - Disk usage monitoring
   - Network I/O statistics

7. **Interactive Features**
   - Data collection details modal
   - Service logs viewer modal
   - Refresh controls for real-time updates

### UI Framework
- **Mantine Components**: Used Mantine UI library for consistency with existing dashboard
- **Responsive Layout**: Grid-based layout for optimal viewing
- **Modern Design**: Cards, badges, alerts, and modals for professional appearance

### Files Modified/Created

1. **`dashboard/layouts/system_health.py`**
   - Complete rewrite using Mantine components
   - Comprehensive layout with monitoring sections
   - Modal dialogs for detailed views

2. **`dashboard/callbacks/system_health.py`**
   - Enhanced callbacks with comprehensive monitoring
   - Real-time status updates
   - Error handling and graceful degradation
   - Integration with database and Redis managers

## Technical Implementation

### Real-time Monitoring Architecture
```python
# Status Update Flow
Interval Component (30s) → Callbacks → Status Checkers → UI Updates
```

### Status Checking Functions
- `_get_data_collection_quick_status()` - Service running detection
- `_get_database_quick_status()` - Database connectivity
- `_get_redis_quick_status()` - Redis connectivity  
- `_get_performance_quick_status()` - System metrics

### Detailed Monitoring Functions
- `_get_data_collection_service_status()` - Service details
- `_get_data_collection_metrics()` - Collection statistics
- `_get_database_status()` & `_get_database_statistics()` - DB health
- `_get_redis_status()` & `_get_redis_statistics()` - Redis health
- `_get_system_performance_metrics()` - System performance

### Error Handling
- Graceful degradation when services are unavailable
- User-friendly error messages with troubleshooting hints
- Fallback status indicators for unknown states

## Integration Points

### Database Integration
- Uses `DatabaseManager` for connection testing
- Queries `market_data` table for collection statistics
- Monitors database performance metrics

### Redis Integration  
- Uses `RedisManager` for connection verification
- Retrieves Redis server information and statistics
- Monitors memory usage and client connections

### System Integration
- Uses `psutil` for system performance monitoring
- Process detection for data collection service
- Resource utilization tracking

## Usage

### Dashboard Access
1. Navigate to "⚙️ System Health" tab in the main dashboard
2. View real-time status cards at the top
3. Explore detailed monitoring sections below

### Service Controls
- **Refresh Status**: Manually refresh data collection status
- **View Details**: Open modal with comprehensive service information
- **View Logs**: Access service logs in scrollable modal

### Status Indicators
- 🟢 **Green**: Healthy/Connected/Good performance
- 🟡 **Yellow**: Warning/Checking/Moderate usage
- 🔴 **Red**: Error/Disconnected/High usage
- ❓ **Gray**: Unknown status

## Future Enhancements

### Planned Improvements (Section 3.7)
1. **Real-time Updates via Redis**: Replace polling with Redis pub/sub
2. **Advanced Metrics**: Historical performance trends
3. **Alerting System**: Notifications for critical issues
4. **Service Management**: Start/stop controls for data collection

### Integration with Data Collection Service
- Real-time collector health reporting
- Performance metrics streaming
- Service configuration management
- Log aggregation and filtering

## Testing

### Manual Testing
1. **Service Detection**: Start/stop data collection service to verify detection
2. **Database Connectivity**: Test with database running/stopped
3. **Redis Connectivity**: Test with Redis running/stopped
4. **Performance Monitoring**: Verify metrics under different system loads

### Integration Testing
- Database manager integration
- Redis manager integration  
- System metrics accuracy
- Error handling scenarios

## Dependencies

### UI Framework
- `dash-mantine-components` - Modern UI components
- `dash` - Core dashboard framework
- `plotly` - Charts and visualizations

### System Monitoring
- `psutil` - System performance metrics
- `subprocess` - Process management
- `datetime` - Time handling

### Database/Redis
- `database.connection.DatabaseManager` - Database operations
- `database.redis_manager.RedisManager` - Redis operations

## Troubleshooting

### Common Issues

1. **"Service Stopped" Status**
   - Solution: Run `python scripts/start_data_collection.py`

2. **Database Connection Failed**
   - Check Docker containers: `docker-compose ps`
   - Verify database configuration in `.env`

3. **Redis Connection Failed**
   - Ensure Redis container is running
   - Check Redis configuration

4. **Performance Metrics Unavailable**
   - Usually permissions issue on system metrics
   - Check if `psutil` has necessary permissions

### Logs and Debugging
- Check dashboard logs for callback errors
- Use browser developer tools for frontend issues
- Monitor system logs for resource issues

## Documentation Updates

### Files Updated
- `tasks/tasks-crypto-bot-prd.md` - Marked Task 3.5 as completed
- Added this documentation file

### Next Task
Ready to proceed with **Task 3.6**: Build simple data analysis tools (volume analysis, price movement statistics)