f09864d61b
- Added `StrategyRun` and `StrategySignal` models to track strategy execution sessions and generated signals, respectively, ensuring a clear separation from live trading data. - Introduced `StrategyRepository` for managing database operations related to strategy runs and signals, including methods for creating, updating, and retrieving strategy data. - Updated `DatabaseOperations` to integrate the new repository, enhancing the overall architecture and maintaining consistency with existing database access patterns. - Enhanced documentation to reflect the new database schema and repository functionalities, ensuring clarity for future development and usage. These changes establish a robust foundation for strategy analysis and backtesting, aligning with project goals for modularity, performance, and maintainability.
707 lines
22 KiB
Markdown
707 lines
22 KiB
Markdown
# Database Operations Documentation
|
|
|
|
## Overview
|
|
|
|
The Database Operations module (`database/operations.py`) provides a clean, centralized interface for all database interactions using the **Repository Pattern**. This approach abstracts SQL complexity from business logic, ensuring maintainable, testable, and consistent database operations across the entire application.
|
|
|
|
## Key Benefits
|
|
|
|
### 🏗️ **Clean Architecture**
|
|
- **Repository Pattern**: Separates data access logic from business logic
|
|
- **Centralized Operations**: All database interactions go through well-defined APIs
|
|
- **No Raw SQL**: Business logic never contains direct SQL queries
|
|
- **Consistent Interface**: Standardized methods across all database operations
|
|
|
|
### 🛡️ **Reliability & Safety**
|
|
- **Automatic Transaction Management**: Sessions and commits handled automatically
|
|
- **Error Handling**: Custom exceptions with proper context
|
|
- **Connection Pooling**: Efficient database connection management
|
|
- **Session Cleanup**: Automatic session management and cleanup
|
|
|
|
### 🔧 **Maintainability**
|
|
- **Easy Testing**: Repository methods can be easily mocked for testing
|
|
- **Database Agnostic**: Can change database implementations without affecting business logic
|
|
- **Type Safety**: Full type hints for better IDE support and error detection
|
|
- **Logging Integration**: Built-in logging for monitoring and debugging
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌───────────────────────────────────────────────────────────────────┐
|
|
│ DatabaseOperations │
|
|
│ ┌───────────────────────────────────────────────────────────┐ │
|
|
│ │ Health Check & Stats │ │
|
|
│ │ • Connection health monitoring │ │
|
|
│ │ • Database statistics │ │
|
|
│ │ • Performance metrics │ │
|
|
│ └───────────────────────────────────────────────────────────┘ │
|
|
│ │ │
|
|
│ ┌─────────────────┐ ┌─────────────────┐ ┌──────────────┐ ┌───────────────────┐ │
|
|
│ │MarketDataRepo │ │RawTradeRepo │ │ BotRepo │ │ StrategyRepo │ │
|
|
│ │ │ │ │ │ │ │ │ │
|
|
│ │ • upsert_candle │ │ • insert_data │ │ • add │ │ • create_run │ │
|
|
│ │ • get_candles │ │ • get_trades │ │ • get_by_id │ │ • store_signals │ │
|
|
│ │ • get_latest │ │ • raw_websocket │ │ • update/delete│ │ • get_signals/stats │ │
|
|
│ └─────────────────┘ └─────────────────┘ └──────────────┘ └───────────────────┘ │
|
|
└───────────────────────────────────────────────────────────────────┘
|
|
│
|
|
┌─────────────────┐
|
|
│ BaseRepository │
|
|
│ │
|
|
│ • Session Mgmt │
|
|
│ • Error Logging │
|
|
│ • DB Connection │
|
|
└─────────────────┘
|
|
```
|
|
|
|
## Quick Start
|
|
|
|
### Basic Usage
|
|
|
|
```python
|
|
from database.operations import get_database_operations
|
|
from data.common.data_types import OHLCVCandle
|
|
from datetime import datetime, timezone
|
|
|
|
# Get the database operations instance (singleton)
|
|
db = get_database_operations()
|
|
|
|
# Check database health
|
|
if not db.health_check():
|
|
print("Database connection issue!")
|
|
return
|
|
|
|
# Store a candle
|
|
candle = OHLCVCandle(
|
|
exchange="okx",
|
|
symbol="BTC-USDT",
|
|
timeframe="5s",
|
|
open=50000.0,
|
|
high=50100.0,
|
|
low=49900.0,
|
|
close=50050.0,
|
|
volume=1.5,
|
|
trade_count=25,
|
|
start_time=datetime(2024, 1, 1, 12, 0, 0, tzinfo=timezone.utc),
|
|
end_time=datetime(2024, 1, 1, 12, 0, 5, tzinfo=timezone.utc)
|
|
)
|
|
|
|
# Store candle (with duplicate handling)
|
|
success = db.market_data.upsert_candle(candle, force_update=False)
|
|
if success:
|
|
print("Candle stored successfully!")
|
|
```
|
|
|
|
### With Data Collectors
|
|
|
|
```python
|
|
import asyncio
|
|
from data.exchanges.okx import OKXCollector
|
|
from data.base_collector import DataType
|
|
from database.operations import get_database_operations
|
|
|
|
async def main():
|
|
# Initialize database operations
|
|
db = get_database_operations()
|
|
|
|
# The collector automatically uses the database operations module
|
|
collector = OKXCollector(
|
|
symbols=['BTC-USDT'],
|
|
data_types=[DataType.TRADE],
|
|
store_raw_data=True, # Stores raw WebSocket data
|
|
force_update_candles=False # Ignore duplicate candles
|
|
)
|
|
|
|
await collector.start()
|
|
await asyncio.sleep(60) # Collect for 1 minute
|
|
await collector.stop()
|
|
|
|
# Check statistics
|
|
stats = db.get_stats()
|
|
print(f"Total bots: {stats['bot_count']}")
|
|
print(f"Total candles: {stats['candle_count']}")
|
|
print(f"Total raw trades: {stats['raw_trade_count']}")
|
|
|
|
asyncio.run(main())
|
|
```
|
|
|
|
## API Reference
|
|
|
|
### DatabaseOperations
|
|
|
|
Main entry point for all database operations.
|
|
|
|
#### Methods
|
|
|
|
##### `health_check() -> bool`
|
|
Test database connection health.
|
|
|
|
```python
|
|
db = get_database_operations()
|
|
if db.health_check():
|
|
print("✅ Database is healthy")
|
|
else:
|
|
print("❌ Database connection issues")
|
|
```
|
|
|
|
##### `get_stats() -> Dict[str, Any]`
|
|
Get comprehensive database statistics.
|
|
|
|
```python
|
|
stats = db.get_stats()
|
|
print(f"Bots: {stats['bot_count']:,}")
|
|
print(f"Candles: {stats['candle_count']:,}")
|
|
print(f"Raw trades: {stats['raw_trade_count']:,}")
|
|
print(f"Strategy runs: {stats['strategy_runs_count']:,}")
|
|
print(f"Strategy signals: {stats['strategy_signals_count']:,}")
|
|
print(f"Health: {stats['healthy']}")
|
|
```
|
|
|
|
### MarketDataRepository
|
|
|
|
Repository for `market_data` table operations (candles/OHLCV data).
|
|
|
|
#### Methods
|
|
|
|
##### `upsert_candle(candle: OHLCVCandle, force_update: bool = False) -> bool`
|
|
|
|
Store or update candle data with configurable duplicate handling.
|
|
|
|
**Parameters:**
|
|
- `candle`: OHLCVCandle object to store
|
|
- `force_update`: If True, overwrites existing data; if False, ignores duplicates
|
|
|
|
**Returns:** True if successful, False otherwise
|
|
|
|
**Duplicate Handling:**
|
|
- `force_update=False`: Uses `ON CONFLICT DO NOTHING` (preserves existing candles)
|
|
- `force_update=True`: Uses `ON CONFLICT DO UPDATE SET` (overwrites existing candles)
|
|
|
|
```python
|
|
# Store new candle, ignore if duplicate exists
|
|
db.market_data.upsert_candle(candle, force_update=False)
|
|
|
|
# Store candle, overwrite if duplicate exists
|
|
db.market_data.upsert_candle(candle, force_update=True)
|
|
```
|
|
|
|
##### `get_candles(symbol: str, timeframe: str, start_time: datetime, end_time: datetime, exchange: str = "okx") -> List[Dict[str, Any]]`
|
|
|
|
Retrieve historical candle data.
|
|
|
|
```python
|
|
from datetime import datetime, timezone
|
|
|
|
candles = db.market_data.get_candles(
|
|
symbol="BTC-USDT",
|
|
timeframe="5s",
|
|
start_time=datetime(2024, 1, 1, 12, 0, 0, tzinfo=timezone.utc),
|
|
end_time=datetime(2024, 1, 1, 13, 0, 0, tzinfo=timezone.utc),
|
|
exchange="okx"
|
|
)
|
|
|
|
for candle in candles:
|
|
print(f"{candle['timestamp']}: O={candle['open']} H={candle['high']} L={candle['low']} C={candle['close']}")
|
|
```
|
|
|
|
##### `get_latest_candle(symbol: str, timeframe: str, exchange: str = "okx") -> Optional[Dict[str, Any]]`
|
|
|
|
Get the most recent candle for a symbol/timeframe combination.
|
|
|
|
```python
|
|
latest = db.market_data.get_latest_candle("BTC-USDT", "5s")
|
|
if latest:
|
|
print(f"Latest 5s candle: {latest['close']} at {latest['timestamp']}")
|
|
else:
|
|
print("No candles found")
|
|
```
|
|
|
|
### BotRepository
|
|
|
|
Repository for `bots` table operations.
|
|
|
|
#### Methods
|
|
|
|
##### `add(bot_data: Dict[str, Any]) -> Bot`
|
|
|
|
Adds a new bot to the database.
|
|
|
|
**Parameters:**
|
|
- `bot_data`: Dictionary containing the bot's attributes (`name`, `strategy_name`, etc.)
|
|
|
|
**Returns:** The newly created `Bot` object.
|
|
|
|
```python
|
|
from decimal import Decimal
|
|
|
|
bot_data = {
|
|
"name": "MyTestBot",
|
|
"strategy_name": "SimpleMACD",
|
|
"symbol": "BTC-USDT",
|
|
"timeframe": "1h",
|
|
"status": "inactive",
|
|
"virtual_balance": Decimal("10000"),
|
|
}
|
|
new_bot = db.bots.add(bot_data)
|
|
print(f"Added bot with ID: {new_bot.id}")
|
|
```
|
|
|
|
##### `get_by_id(bot_id: int) -> Optional[Bot]`
|
|
|
|
Retrieves a bot by its unique ID.
|
|
|
|
```python
|
|
bot = db.bots.get_by_id(1)
|
|
if bot:
|
|
print(f"Found bot: {bot.name}")
|
|
```
|
|
|
|
##### `get_by_name(name: str) -> Optional[Bot]`
|
|
|
|
Retrieves a bot by its unique name.
|
|
|
|
```python
|
|
bot = db.bots.get_by_name("MyTestBot")
|
|
if bot:
|
|
print(f"Found bot with ID: {bot.id}")
|
|
```
|
|
|
|
##### `update(bot_id: int, update_data: Dict[str, Any]) -> Optional[Bot]`
|
|
|
|
Updates an existing bot's attributes.
|
|
|
|
```python
|
|
from datetime import datetime, timezone
|
|
|
|
update_payload = {"status": "active", "last_heartbeat": datetime.now(timezone.utc)}
|
|
updated_bot = db.bots.update(1, update_payload)
|
|
if updated_bot:
|
|
print(f"Bot status updated to: {updated_bot.status}")
|
|
```
|
|
|
|
##### `delete(bot_id: int) -> bool`
|
|
|
|
Deletes a bot from the database.
|
|
|
|
**Returns:** `True` if deletion was successful, `False` otherwise.
|
|
|
|
```python
|
|
success = db.bots.delete(1)
|
|
if success:
|
|
print("Bot deleted successfully.")
|
|
```
|
|
|
|
### RawTradeRepository
|
|
|
|
Repository for `raw_trades` table operations (raw WebSocket data).
|
|
|
|
#### Methods
|
|
|
|
##### `insert_market_data_point(data_point: MarketDataPoint) -> bool`
|
|
|
|
Store raw market data from WebSocket streams.
|
|
|
|
```python
|
|
from data.base_collector import MarketDataPoint, DataType
|
|
from datetime import datetime, timezone
|
|
|
|
data_point = MarketDataPoint(
|
|
exchange="okx",
|
|
symbol="BTC-USDT",
|
|
timestamp=datetime.now(timezone.utc),
|
|
data_type=DataType.TRADE,
|
|
data={"price": 50000, "size": 0.1, "side": "buy"}
|
|
)
|
|
|
|
success = db.raw_trades.insert_market_data_point(data_point)
|
|
```
|
|
|
|
##### `insert_raw_websocket_data(exchange: str, symbol: str, data_type: str, raw_data: Dict[str, Any], timestamp: Optional[datetime] = None) -> bool`
|
|
|
|
Store raw WebSocket data for debugging purposes.
|
|
|
|
```python
|
|
db.raw_trades.insert_raw_websocket_data(
|
|
exchange="okx",
|
|
symbol="BTC-USDT",
|
|
data_type="raw_trade",
|
|
raw_data={"instId": "BTC-USDT", "px": "50000", "sz": "0.1"},
|
|
timestamp=datetime.now(timezone.utc)
|
|
)
|
|
```
|
|
|
|
##### `get_raw_trades(symbol: str, data_type: str, start_time: datetime, end_time: datetime, exchange: str = "okx", limit: Optional[int] = None) -> List[Dict[str, Any]]`
|
|
|
|
Retrieve raw trade data for analysis.
|
|
|
|
```python
|
|
trades = db.raw_trades.get_raw_trades(
|
|
symbol="BTC-USDT",
|
|
data_type="trade",
|
|
start_time=datetime(2024, 1, 1, 12, 0, 0, tzinfo=timezone.utc),
|
|
end_time=datetime(2024, 1, 1, 13, 0, 0, tzinfo=timezone.utc),
|
|
limit=1000
|
|
)
|
|
```
|
|
|
|
##### `cleanup_old_raw_data(days_to_keep: int = 7) -> int`
|
|
|
|
Clean up old raw data to prevent table bloat.
|
|
|
|
**Parameters:**
|
|
- `days_to_keep`: Number of days to retain raw data records.
|
|
|
|
**Returns:** The number of records deleted.
|
|
|
|
```python
|
|
# Clean up raw data older than 14 days
|
|
deleted_count = db.raw_trades.cleanup_old_raw_data(days_to_keep=14)
|
|
print(f"Deleted {deleted_count} old raw data records.")
|
|
```
|
|
|
|
##### `get_raw_data_stats() -> Dict[str, Any]`
|
|
|
|
Get statistics about raw data storage.
|
|
|
|
**Returns:** A dictionary with statistics like total records, table size, etc.
|
|
|
|
```python
|
|
raw_stats = db.raw_trades.get_raw_data_stats()
|
|
print(f"Raw Trades Table Size: {raw_stats.get('table_size')}")
|
|
print(f"Total Raw Records: {raw_stats.get('total_records')}")
|
|
```
|
|
|
|
### StrategyRepository
|
|
|
|
Repository for `strategy_signals` and `strategy_runs` table operations.
|
|
|
|
#### Methods
|
|
|
|
##### `create_strategy_run(run_data: Dict[str, Any]) -> StrategyRun`
|
|
|
|
Create a new strategy run session.
|
|
|
|
**Parameters:**
|
|
- `run_data`: Dictionary containing run information (`strategy_name`, `symbol`, `timeframe`, `start_time`, `status`, `config`, `run_metadata`)
|
|
|
|
**Returns:** The newly created `StrategyRun` object.
|
|
|
|
```python
|
|
from datetime import datetime, timezone
|
|
|
|
run_data = {
|
|
"strategy_name": "EMA Crossover Backtest",
|
|
"symbol": "BTC-USDT",
|
|
"timeframe": "1h",
|
|
"start_time": datetime(2023, 1, 1, tzinfo=timezone.utc),
|
|
"status": "running",
|
|
"config": {"fast_ema": 12, "slow_ema": 26},
|
|
"run_metadata": {"backtest_period": "2023-Q1"}
|
|
}
|
|
new_run = db.strategies.create_strategy_run(run_data)
|
|
print(f"Created strategy run with ID: {new_run.id}")
|
|
```
|
|
|
|
##### `get_strategy_run_by_id(run_id: int) -> Optional[StrategyRun]`
|
|
|
|
Get a strategy run by its ID.
|
|
|
|
```python
|
|
run = db.strategies.get_strategy_run_by_id(1)
|
|
if run:
|
|
print(f"Found strategy run: {run.strategy_name} ({run.status})")
|
|
```
|
|
|
|
##### `update_strategy_run(run_id: int, update_data: Dict[str, Any]) -> Optional[StrategyRun]`
|
|
|
|
Update a strategy run's information.
|
|
|
|
```python
|
|
update_payload = {"status": "completed", "total_signals": 150}
|
|
updated_run = db.strategies.update_strategy_run(1, update_payload)
|
|
if updated_run:
|
|
print(f"Strategy run status updated to: {updated_run.status}")
|
|
```
|
|
|
|
##### `complete_strategy_run(run_id: int, total_signals: int) -> bool`
|
|
|
|
Mark a strategy run as completed.
|
|
|
|
**Parameters:**
|
|
- `run_id`: The ID of the strategy run to complete.
|
|
- `total_signals`: The total number of signals generated during the run.
|
|
|
|
**Returns:** `True` if successful, `False` otherwise.
|
|
|
|
```python
|
|
success = db.strategies.complete_strategy_run(1, 150)
|
|
if success:
|
|
print("Strategy run marked as completed.")
|
|
```
|
|
|
|
##### `store_strategy_signals(run_id: int, strategy_results: List[StrategyResult]) -> int`
|
|
|
|
Store multiple strategy signals from strategy results.
|
|
|
|
**Parameters:**
|
|
- `run_id`: The strategy run ID these signals belong to.
|
|
- `strategy_results`: List of `StrategyResult` objects containing signals.
|
|
|
|
**Returns:** Number of signals stored.
|
|
|
|
```python
|
|
from strategies.data_types import StrategyResult, StrategySignal, SignalType
|
|
from datetime import datetime, timezone
|
|
from decimal import Decimal
|
|
|
|
signal1 = StrategySignal(
|
|
timestamp=datetime.now(timezone.utc),
|
|
symbol="BTC-USDT",
|
|
timeframe="1h",
|
|
signal_type=SignalType.BUY,
|
|
price=Decimal("50000.0"),
|
|
confidence=Decimal("0.9")
|
|
)
|
|
result1 = StrategyResult(
|
|
timestamp=datetime.now(timezone.utc),
|
|
symbol="BTC-USDT",
|
|
timeframe="1h",
|
|
strategy_name="EMA Crossover",
|
|
signals=[signal1],
|
|
indicators_used={"ema_fast": 49900.0, "ema_slow": 50100.0}
|
|
)
|
|
|
|
signals_stored = db.strategies.store_strategy_signals(new_run.id, [result1])
|
|
print(f"Stored {signals_stored} signals.")
|
|
```
|
|
|
|
##### `get_strategy_signals(run_id: Optional[int] = None, strategy_name: Optional[str] = None, symbol: Optional[str] = None, timeframe: Optional[str] = None, start_time: Optional[datetime] = None, end_time: Optional[datetime] = None, signal_type: Optional[str] = None, limit: Optional[int] = None) -> List[StrategySignal]`
|
|
|
|
Retrieve strategy signals with flexible filtering.
|
|
|
|
**Parameters:**
|
|
- `run_id`: Filter by strategy run ID.
|
|
- `strategy_name`: Filter by strategy name.
|
|
- `symbol`: Filter by trading symbol.
|
|
- `timeframe`: Filter by timeframe.
|
|
- `start_time`: Filter signals after this time.
|
|
- `end_time`: Filter signals before this time.
|
|
- `signal_type`: Filter by signal type (e.g., "buy", "sell", "entry_long").
|
|
- `limit`: Maximum number of signals to return.
|
|
|
|
**Returns:** List of `StrategySignal` objects.
|
|
|
|
```python
|
|
signals = db.strategies.get_strategy_signals(
|
|
run_id=new_run.id,
|
|
signal_type="buy",
|
|
limit=10
|
|
)
|
|
for signal in signals:
|
|
print(f"Signal: {signal.signal_type} at {signal.price} on {signal.timestamp}")
|
|
```
|
|
|
|
##### `get_strategy_signal_stats(run_id: Optional[int] = None) -> Dict[str, Any]`
|
|
|
|
Get statistics about strategy signals.
|
|
|
|
**Parameters:**
|
|
- `run_id`: Optional run ID to get stats for a specific run.
|
|
|
|
**Returns:** Dictionary with statistics like total signals, counts by type, average confidence.
|
|
|
|
```python
|
|
signal_stats = db.strategies.get_strategy_signal_stats(run_id=new_run.id)
|
|
print(f"Total Signals: {signal_stats['total_signals']}")
|
|
print(f"Signal Counts: {signal_stats['signal_counts']}")
|
|
print(f"Average Confidence: {signal_stats['average_confidence']:.2f}")
|
|
```
|
|
|
|
##### `cleanup_old_strategy_data(days_to_keep: int = 30) -> Dict[str, int]`
|
|
|
|
Clean up old strategy signals and runs.
|
|
|
|
**Parameters:**
|
|
- `days_to_keep`: Number of days to retain completed strategy runs and their signals.
|
|
|
|
**Returns:** Dictionary with counts of deleted runs and signals.
|
|
|
|
```python
|
|
deleted_info = db.strategies.cleanup_old_strategy_data(days_to_keep=60)
|
|
print(f"Deleted {deleted_info['deleted_runs']} old strategy runs.")
|
|
```
|
|
|
|
## Error Handling
|
|
|
|
The database operations module includes comprehensive error handling with custom exceptions.
|
|
|
|
### DatabaseOperationError
|
|
|
|
Custom exception for database operation failures.
|
|
|
|
```python
|
|
from database.operations import DatabaseOperationError
|
|
|
|
try:
|
|
db.market_data.upsert_candle(candle)
|
|
except DatabaseOperationError as e:
|
|
logger.error(f"Database operation failed: {e}")
|
|
# Handle the error appropriately
|
|
```
|
|
|
|
### Best Practices
|
|
|
|
1. **Always Handle Exceptions**: Wrap database operations in try-catch blocks
|
|
2. **Check Health First**: Use `health_check()` before critical operations
|
|
3. **Monitor Performance**: Use `get_stats()` to monitor database growth
|
|
4. **Use Appropriate Repositories**: Use `market_data` for candles, `raw_trades` for raw data
|
|
5. **Handle Duplicates Appropriately**: Choose the right `force_update` setting
|
|
|
|
## Configuration
|
|
|
|
### Force Update Behavior
|
|
|
|
The `force_update_candles` parameter in collectors controls duplicate handling:
|
|
|
|
```python
|
|
# In OKX collector configuration
|
|
collector = OKXCollector(
|
|
symbols=['BTC-USDT'],
|
|
force_update_candles=False # Default: ignore duplicates
|
|
)
|
|
|
|
# Or enable force updates
|
|
collector = OKXCollector(
|
|
symbols=['BTC-USDT'],
|
|
force_update_candles=True # Overwrite existing candles
|
|
)
|
|
```
|
|
|
|
### Logging Integration
|
|
|
|
Database operations automatically integrate with the application's logging system:
|
|
|
|
```python
|
|
import logging
|
|
from database.operations import get_database_operations
|
|
|
|
logger = logging.getLogger(__name__)
|
|
db = get_database_operations(logger)
|
|
|
|
# All database operations will now log through your logger
|
|
db.market_data.upsert_candle(candle) # Logs: "Stored candle: BTC-USDT 5s at ..."
|
|
```
|
|
|
|
## Migration from Direct SQL
|
|
|
|
If you have existing code using direct SQL, here's how to migrate:
|
|
|
|
### Before (Direct SQL - ❌ Don't do this)
|
|
|
|
```python
|
|
# OLD WAY - direct SQL queries
|
|
from database.connection import get_db_manager
|
|
from sqlalchemy import text
|
|
|
|
db_manager = get_db_manager()
|
|
with db_manager.get_session() as session:
|
|
session.execute(text("""
|
|
INSERT INTO market_data (exchange, symbol, timeframe, ...)
|
|
VALUES (:exchange, :symbol, :timeframe, ...)
|
|
"""), {'exchange': 'okx', 'symbol': 'BTC-USDT', ...})
|
|
session.commit()
|
|
```
|
|
|
|
### After (Repository Pattern - ✅ Correct way)
|
|
|
|
```python
|
|
# NEW WAY - using repository pattern
|
|
from database.operations import get_database_operations
|
|
from data.common.data_types import OHLCVCandle
|
|
|
|
db = get_database_operations()
|
|
candle = OHLCVCandle(...) # Create candle object
|
|
success = db.market_data.upsert_candle(candle)
|
|
```
|
|
|
|
The entire repository layer has been standardized to use the SQLAlchemy ORM internally, ensuring a consistent, maintainable, and database-agnostic approach. Raw SQL is avoided in favor of type-safe ORM queries.
|
|
|
|
## Performance Considerations
|
|
|
|
### Connection Pooling
|
|
|
|
The database operations module automatically manages connection pooling through the underlying `DatabaseManager`.
|
|
|
|
### Batch Operations
|
|
|
|
For high-throughput scenarios, consider batching operations:
|
|
|
|
```python
|
|
# Store multiple candles efficiently
|
|
candles = [candle1, candle2, candle3, ...]
|
|
|
|
for candle in candles:
|
|
db.market_data.upsert_candle(candle)
|
|
```
|
|
|
|
### Monitoring
|
|
|
|
Monitor database performance using the built-in statistics:
|
|
|
|
```python
|
|
import time
|
|
|
|
# Monitor database load
|
|
while True:
|
|
stats = db.get_stats()
|
|
print(f"Candles: {stats['candle_count']:,}, Health: {stats['healthy']}")
|
|
time.sleep(30)
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
#### 1. Connection Errors
|
|
```python
|
|
if not db.health_check():
|
|
logger.error("Database connection failed - check connection settings")
|
|
```
|
|
|
|
#### 2. Duplicate Key Errors
|
|
```python
|
|
# Use force_update=False to ignore duplicates
|
|
db.market_data.upsert_candle(candle, force_update=False)
|
|
```
|
|
|
|
#### 3. Transaction Errors
|
|
The repository automatically handles session management, but if you encounter issues:
|
|
```python
|
|
try:
|
|
db.market_data.upsert_candle(candle)
|
|
except DatabaseOperationError as e:
|
|
logger.error(f"Transaction failed: {e}")
|
|
```
|
|
|
|
### Debug Mode
|
|
|
|
Enable database query logging for debugging:
|
|
|
|
```python
|
|
# Set environment variable
|
|
import os
|
|
os.environ['DEBUG'] = 'true'
|
|
|
|
# This will log all SQL queries
|
|
db = get_database_operations()
|
|
```
|
|
|
|
## Related Documentation
|
|
|
|
- **[Database Connection](../architecture/database.md)** - Connection pooling and configuration
|
|
- **[Data Collectors](data_collectors.md)** - How collectors use database operations
|
|
- **[Architecture Overview](../architecture/architecture.md)** - System design patterns
|
|
|
|
---
|
|
|
|
*This documentation covers the repository pattern implementation in `database/operations.py`. For database schema details, see the [Architecture Documentation](../architecture/).* |