orderflow_backtest/docs/modules/main.md

# Module: main

## Purpose
The `main` module provides the command-line interface (CLI) orchestration for the orderflow backtest system. It handles database discovery, process management, and coordinates the streaming pipeline with the visualization frontend using Typer for argument parsing.

## Public Interface

### Functions
- `main(instrument: str, start_date: str, end_date: str, window_seconds: int = 60) -> None`: Primary CLI entrypoint
- `discover_databases(instrument: str, start_date: str, end_date: str) -> list[Path]`: Find matching database files
- `launch_visualizer() -> subprocess.Popen | None`: Start Dash application in separate process

### CLI Arguments
- `instrument`: Trading pair identifier (e.g., "BTC-USDT")
- `start_date`: Start date in YYYY-MM-DD format (UTC)
- `end_date`: End date in YYYY-MM-DD format (UTC)
- `--window-seconds`: OHLC aggregation window size (default: 60)

## Usage Examples

### Command Line Usage
```bash
# Basic usage with default 60-second windows
uv run python main.py BTC-USDT 2025-01-01 2025-01-31

# Custom window size
uv run python main.py ETH-USDT 2025-02-01 2025-02-28 --window-seconds 30

# Single day processing
uv run python main.py SOL-USDT 2025-03-15 2025-03-15
```

### Programmatic Usage
```python
from main import main, discover_databases

# Run processing pipeline
main("BTC-USDT", "2025-01-01", "2025-01-31", window_seconds=120)

# Discover available databases
db_files = discover_databases("ETH-USDT", "2025-02-01", "2025-02-28")
print(f"Found {len(db_files)} database files")
```

## Dependencies

### Internal
- `db_interpreter.DBInterpreter`: Database streaming
- `ohlc_processor.OHLCProcessor`: Trade aggregation and orderbook processing
- `viz_io`: Data clearing functions

### External
- `typer`: CLI framework and argument parsing
- `subprocess`: Process management for visualization
- `pathlib`: File and directory operations
- `datetime`: Date parsing and validation
- `logging`: Operational logging
- `sys`: Exit code management

## Database Discovery Logic

### File Pattern Matching
```python
# Expected directory structure
../data/OKX/{instrument}/{date}/

# Example paths
../data/OKX/BTC-USDT/2025-01-01/trades.db
../data/OKX/ETH-USDT/2025-02-15/trades.db
```

### Discovery Algorithm
1. Parse start and end dates to datetime objects
2. Iterate through date range (inclusive)
3. Construct expected path for each date
4. Verify file existence and readability
5. Return sorted list of valid database paths

## Process Orchestration

### Visualization Process Management
```python
# Launch Dash app in separate process
viz_process = subprocess.Popen([
    "uv", "run", "python", "app.py"
], cwd=project_root)

# Process management
try:
    # Main processing loop
    process_databases(db_files)
finally:
    # Cleanup visualization process
    if viz_process:
        viz_process.terminate()
        viz_process.wait(timeout=5)
```

### Data Processing Pipeline
1. **Initialize**: Clear existing data files
2. **Launch**: Start visualization process
3. **Stream**: Process each database sequentially
4. **Aggregate**: Generate OHLC bars and depth snapshots
5. **Cleanup**: Terminate visualization and finalize

## Error Handling

### Database Access Errors
- **File not found**: Log warning and skip missing databases
- **Permission denied**: Log error and exit with status code 1
- **Corruption**: Log error for specific database and continue with next

### Process Management Errors
- **Visualization startup failure**: Log error but continue processing
- **Process termination**: Graceful shutdown with timeout
- **Resource cleanup**: Ensure child processes are terminated

### Date Validation
- **Invalid format**: Clear error message with expected format
- **Invalid range**: End date must be >= start date
- **Future dates**: Warning for dates beyond data availability

## Performance Characteristics

- **Sequential processing**: Databases processed one at a time
- **Memory efficient**: Streaming approach prevents loading entire datasets
- **Process isolation**: Visualization runs independently
- **Resource cleanup**: Automatic process termination on exit

## Testing

Run module tests:
```bash
uv run pytest test_main.py -v
```

Test coverage includes:
- Database discovery logic
- Date parsing and validation
- Process management
- Error handling scenarios
- CLI argument validation

## Configuration

### Default Settings
- **Data directory**: `../data/OKX` (relative to project root)
- **Visualization command**: `uv run python app.py`
- **Window size**: 60 seconds
- **Process timeout**: 5 seconds for termination

### Environment Variables
- **DATA_PATH**: Override default data directory
- **VISUALIZATION_PORT**: Override Dash port (requires app.py modification)

## Known Issues

- Assumes specific directory structure under `../data/OKX`
- No validation of database schema compatibility
- Limited error recovery for process management
- No progress indication for large datasets

## Development Notes

- Uses Typer for modern CLI interface
- Subprocess management compatible with Unix/Windows
- Logging configured for both development and production use
- Exit codes follow Unix conventions (0=success, 1=error)