orderflow_backtest/prd-obi-cvd-metrics.md

PRD: OBI and CVD Metrics Integration

Introduction/Overview

This feature integrates Order Book Imbalance (OBI) and Cumulative Volume Delta (CVD) calculations into the orderflow backtest system. Currently, the system stores all snapshots in memory during processing, which consumes excessive memory. The goal is to compute OBI and CVD metrics during the build_booktick_from_db execution, store these metrics persistently in the database, and visualize them alongside OHLC and volume data.

Goals

1. Memory Optimization: Reduce memory usage by storing only essential data (OBI/CVD metrics, best bid/ask) instead of full snapshot history
2. Metric Calculation: Implement per-snapshot OBI and CVD calculations with maximum granularity
3. Persistent Storage: Store calculated metrics in the database to avoid recalculation
4. Enhanced Visualization: Display OBI and CVD curves beneath volume graphs with shared time axis
5. Incremental CVD: Support incremental CVD calculation that can be reset at user-defined points

User Stories

1. As a trader, I want to see OBI and CVD metrics calculated for each orderbook snapshot so that I can analyze market sentiment with maximum granularity.

2. As a system user, I want metrics to be stored persistently in the database so that I don't need to recalculate them when re-analyzing the same dataset.

3. As a data analyst, I want to visualize OBI and CVD curves alongside OHLC and volume data so that I can correlate price movements with orderbook imbalances and volume deltas.

4. As a performance-conscious user, I want the system to use less memory during processing so that I can analyze larger datasets (months to years of data).

5. As a researcher, I want incremental CVD calculation so that I can track cumulative volume changes from any chosen starting point in my analysis.

Functional Requirements

Database Schema Updates

1. Create metrics table with the following structure:

   CREATE TABLE metrics (
       id INTEGER PRIMARY KEY AUTOINCREMENT,
       snapshot_id INTEGER,
       timestamp TEXT,
       obi REAL,
       cvd REAL,
       best_bid REAL,
       best_ask REAL,
       FOREIGN KEY (snapshot_id) REFERENCES book(id)
   );
   

OBI Calculation

2. Calculate OBI per snapshot using the formula: OBI = (Vb - Va) / (Vb + Va) where:
   • Vb = total volume on bid side
   • Va = total volume on ask side
3. Handle edge cases where Vb + Va = 0 by setting OBI = 0.0
4. Store OBI values in the metrics table for each processed snapshot

CVD Calculation

5. Calculate Volume Delta per timestamp by summing all trades at each snapshot timestamp:
   • Buy trades (side = "buy"): add to positive volume
   • Sell trades (side = "sell"): add to negative volume
   • VD = Buy Volume - Sell Volume
6. Calculate Cumulative Volume Delta as running sum: CVD_t = CVD_{t-1} + VD_t
7. Support CVD reset functionality to allow starting cumulative calculation from any point
8. Handle snapshots with no trades by carrying forward the previous CVD value

Storage System Updates

9. Modify Storage class to integrate metric calculations during build_booktick_from_db
10. Update Book model to store only essential data: OBI/CVD time series and best bid/ask levels
11. Remove full snapshot retention from memory after metric calculation
12. Add metric persistence to SQLite database during processing

Strategy Integration

13. Enhance DefaultStrategy to calculate both OBI and CVD metrics
14. Return time-series data structures compatible with visualization system
15. Integrate metric calculation into the existing on_booktick workflow

Visualization Enhancements

16. Add OBI and CVD plotting to the visualizer beneath volume graphs
17. Implement shared X-axis for time alignment across OHLC, volume, OBI, and CVD charts
18. Support 6-hour bar aggregation as the initial time resolution
19. Use standard line styling for OBI and CVD curves
20. Make time resolution configurable for future flexibility

Non-Goals (Out of Scope)

1. Real-time streaming - This feature focuses on historical data processing
2. Advanced visualization features - Complex styling, indicators, or interactive elements beyond basic line charts
3. Alternative CVD calculation methods - Only implementing the standard buy/sell volume delta approach
4. Multi-threading optimization - Simple sequential processing for initial implementation
5. Data compression - No advanced compression techniques for stored metrics
6. Export functionality - No CSV/JSON export of calculated metrics

Technical Considerations

Database Performance

• Add indexes on metrics.timestamp and metrics.snapshot_id for efficient querying
• Consider batch insertions for metric data to improve write performance

Memory Management

• Process snapshots sequentially and discard after metric calculation
• Maintain only calculated time-series data in memory
• Keep best bid/ask data for potential future analysis needs

Data Integrity

• Ensure metric calculations are atomic with snapshot processing
• Add foreign key constraints to maintain referential integrity
• Implement transaction boundaries for consistent data state

Integration Points

• Modify SQLiteOrderflowRepository to support metrics table operations
• Update Storage._create_snapshots_from_rows to include metric calculation
• Extend Visualizer to handle additional metric data series

Success Metrics

1. Memory Usage Reduction: Achieve at least 70% reduction in peak memory usage during processing
2. Processing Speed: Maintain or improve current processing speed (rows/sec) despite additional calculations
3. Data Accuracy: 100% correlation between manually calculated and stored OBI/CVD values for test datasets
4. Visualization Quality: Successfully display OBI and CVD curves with proper time alignment
5. Storage Efficiency: Metrics table size should be manageable relative to source data (< 20% overhead)

Open Questions

1. Index Strategy: Should we add additional database indexes for time-range queries on metrics?
2. CVD Starting Value: Should CVD start from 0 for each database file, or allow continuation from previous sessions?
3. Error Recovery: How should the system handle partial metric calculations if processing is interrupted?
4. Validation: Do we need validation checks to ensure OBI values stay within [-1, 1] range?
5. Performance Monitoring: Should we add timing metrics to track calculation performance per snapshot?

Implementation Priority

Phase 1: Core Functionality

• Database schema updates
• Basic OBI/CVD calculation
• Metric storage integration

Phase 2: Memory Optimization

• Remove full snapshot retention
• Implement essential data-only storage

Phase 3: Visualization

• Add metric plotting to visualizer
• Implement time axis alignment
• Support 6-hour bar aggregation